ONNX inference plugin for Mesh

Run Indra OCR, CLIP ViT-B/32 image/text similarity, Whisper Tiny transcription and Depth Anything V2 Small using native Rust and ONNX Runtime. Mesh owns model configuration and asset acquisition; the plugin owns preprocessing and inference, with independent on-demand loading and idle eviction.

Start with the model documentation, including a configuration example and HTTP request for every supported model:

• Indra OCR
• CLIP image/text similarity
• Whisper transcription
• Depth estimation
• End-to-end HTTP example: one submission through all four models

For example, with the plugin installed and the companion Mesh managed-model change:

[[plugin]]
name = "onnx"
command = "/absolute/path/to/mesh-onnx"

[[models]]
model = "datalab-to/indra-ocr@bd02167fdf2005a19962cc15188f8496a2c87d3b"
provider = "onnx"
[models.options]
threads = 4
math = true
[models.lifecycle]
load = "on_demand"
idle_timeout_secs = 300
prefetch = true

Run mesh-llm serve, then invoke OCR through Mesh's inference listener:

curl --fail-with-body -H 'Accept: application/json' \
  -F 'model=datalab-to/indra-ocr@bd02167fdf2005a19962cc15188f8496a2c87d3b' -F 'file=@scan.png' \
  http://localhost:9337/mesh/v1/documents/ocr

Mesh downloads the selected compatible asset bundle into its shared Hugging Face cache. With this policy, the plugin creates sessions on first use and drops them after five idle minutes; later requests reuse cached weights. Images need no Pdfium. PDF uploads require a platform Pdfium library: see the Indra guide.

Multipart fields are model, file and the task's own options. The maximum file size is 64 MiB. PDF pages render at 200 DPI, capped at 4,096 pixels per dimension, and are recognized one at a time. Use Accept to select application/json (the default), text/plain, or text/markdown for OCR. Quality weights and more specific media ranges apply; q=0 excludes a representation; equal weights prefer JSON, then plain text, then Markdown. An unacceptable header returns 406 before the document is read, and OCR responses, including errors, carry Vary: Accept. JSON returns the model ID and page-indexed text, markdown, lines with boxes and confidence, and ordered layout blocks; errors use error.message.

For direct clients of older provider versions, /v1/documents/ocr remains an alias. Only on that legacy path, and only when Accept is absent, multipart response_format=json|text|markdown selects the output. Any supplied Accept overrides the body field, including */*; an unacceptable header never falls back to it. On the canonical /mesh/v1/documents/ocr path the field is rejected, and the OpenAI-shaped /v1/audio/transcriptions route accepts response_format=json|text as its protocol defines. Use the canonical path through current Mesh hosts; the provider alias does not add an alias to Mesh ingress.

See configuration for local directories, source overrides, variants, request defaults and lifecycle semantics. Model-specific options are validated before downloads; one task does not inherit another task's controls.

Build and test

Model-free contract tests — negotiation defaults, header precedence, weights, exclusions, 406, Vary, route admission and the legacy alias — run without model weights or Pdfium:

cargo build --release
cargo test --offline
cargo clippy --offline --all-targets -- -D warnings

This implementation requires the companion Mesh model-asset RPC and provider configuration support. Model weights remain outside the binary. The ort crate supplies ONNX Runtime; inspect native library dependencies when packaging. Pdfium is optional for images.

The real-weight suite covers every supported precision, complete math decoding, and encrypted/hybrid PDFs. The in-repository live tests launch Mesh and the plugin to certify cold loading, concurrent requests, idle eviction, cached reload and peer routing. See validation for commands, exact model pins and recorded results. Adapter limitations and response semantics are documented per model.

Indra code is Apache-2.0; its model weights use modified OpenRAIL-M terms. Other model licenses and artifact provenance must be checked at their pinned source revisions before packaging or redistribution.

Migrating the OCR-only plugin

The managed provider uses plugin name onnx, executable mesh-onnx and Pdfium variable ONNX_PDFIUM_LIBRARY. Update existing onnx-ocr / mesh-onnx-ocr configurations and ONNX_OCR_PDFIUM_LIBRARY when installing it. Model selections belong under [[models]] with provider = "onnx"; the Indra selection now pins the immutable commit shown above.