Skip to content
BranchesTags

Repository files navigation

LuxonisEval

License

🌟 Overview

LuxonisEval is a modular framework for evaluating neural network models across multiple inference engines. It supports on-device inference on Luxonis devices (RVC2 and RVC4) through DepthAI, as well as host-side inference through ONNX Runtime, while reporting both quality metrics and throughput or latency performance.

Typical use cases include:

  • Validating PyTorch checkpoint to ONNX exports by comparing an ONNX model against metrics from its original training checkpoint.
  • Measuring conversion impact by comparing host-side ONNX results with the same model compiled for RVC2 or RVC4.
  • Evaluating quantization tradeoffs across different modes such as FP16 and INT8.
  • Detecting regressions in model quality, preprocessing, output parsing, or NNArchive metadata as models and config parameters evolve.
  • Reviewing predictions qualitatively by saving annotated evaluation samples.

The framework follows a registry-based architecture: each pluggable component (engines, dataloaders, parsers, metrics, and visualizers) registers itself automatically. This lets you swap, extend, or add parts of the evaluation pipeline without modifying the core evaluation loop. In practice, adding a new component usually means subclassing the appropriate base class and referencing it by name in the configuration.

✨ Key Features

  • Multiple Inference Engines
  • Dataset Loading
  • Current Evaluation Coverage - The built-in parsers and metrics currently cover classification, bounding box detection, semantic segmentation, instance segmentation, and keypoint evaluation
  • NNArchive-Aware Configuration - Parser metadata and preprocessing hints can be resolved from NNArchive models, including archive-driven overrides when desired
  • Extensible Architecture - The registry-based design powered by AutoRegisterMeta makes it straightforward to add custom engines, parsers, metrics, loaders, and visualizers

🚀 Quick Start

Get started with LuxonisEval in a few steps:

  1. Install the project from source

    pip install .

  2. Install FiftyOne, then download the models and prepare 1,000 COCO images

    pip install fiftyone
bash examples/quickstart/setup.sh

  3. Run the evaluation

    luxonis_eval eval --config examples/quickstart/onnx_config.yaml

This quickstart evaluates a YOLOv6 NNArchive from models.luxonis.com with ONNX Runtime on CPU and does not require Luxonis hardware. If you have an RVC4 device, the same example can evaluate the RVC4 archive on-device. For model placement, dataset parsing, visualization, and device instructions, see examples/quickstart/README.md.

📜 Table Of Contents

🛠️ Installation

LuxonisEval requires Python 3.10 or higher. We recommend using a virtual environment to keep dependencies isolated.

Install from source:

pip install .

This installs the luxonis_eval CLI in your environment.

Developer install:

pip install -e ".[dev]"

📝 Usage

You can use LuxonisEval either from the command line or through the Python API. The CLI is the primary entry point for running evaluations from configuration files.

💻 CLI

The CLI currently exposes the eval command:

luxonis_eval eval --help

Example invocations:

# Run evaluation with a config file
luxonis_eval eval --config path/to/config.yaml

# Run with CLI overrides
luxonis_eval eval \
    --config path/to/config.yaml \
    --dataset-name coco \
    --model-path path/to/model.tar.xz \
    --backend depthai

# Use the ONNX engine
luxonis_eval eval \
    --config path/to/config.yaml \
    --dataset-name coco \
    --model-path path/to/model.onnx \
    --backend onnx

# Specify device IP for RVC4
luxonis_eval eval \
    --config path/to/config.yaml \
    --device-ip 192.168.1.100

🐍 Python API

For one-shot programmatic usage, call eval_run:

from luxonis_eval.__main__ import eval_run
from luxonis_eval.config import EvalConfig

eval_cfg = EvalConfig.get_config(cfg="path/to/config.yaml")
results = eval_run(eval_cfg)

If you need explicit lifecycle control, repeated evaluate() calls after one setup, or direct access to runtime state, use LuxonisEval directly:

from luxonis_eval import LuxonisEval
from luxonis_eval.config import EvalConfig

eval_cfg = EvalConfig.get_config(cfg="path/to/config.yaml")
evaluator = LuxonisEval(eval_cfg)
evaluator.setup()
try:
    results = evaluator.evaluate()
finally:
    evaluator.close()

🏗️ Architecture

The repository is organized around a small set of core component types:

luxonis_eval/
├── config/           # Configuration schema and exports
├── core/             # Evaluation lifecycle orchestration
├── engines/          # Inference engines
├── loaders/          # Dataset loaders
├── metrics/          # Evaluation metrics
├── parsers/          # Model output parsers
├── utils/            # Shared low-level helpers
├── visualizers/      # Result visualization
└── metadata/         # Class mapping files

🧩 Key Base Classes

Base Class Location Purpose
BaseEngine engines/ Abstract inference engine
BaseParser parsers/ Abstract output parser
BaseMetric metrics/ Abstract evaluation metric
BaseEvalLoader loaders/ Abstract dataset loader
BaseVisualizer visualizers/ Abstract result visualizer

All base classes use the AutoRegisterMeta metaclass. Any subclass is registered automatically and becomes available by name in configuration files, with no manual wiring required.

🔄 Evaluation Pipeline

The evaluation loop in LuxonisEval.evaluate() is structured around abstract component interfaces rather than concrete implementations. That design keeps the pipeline modular and makes engine-specific or model-specific components easy to replace.

┌────────────┐     ┌─────────────┐     ┌─────────────┐
│ DataLoader │────▶│    Engine   │────▶│   Parser    │
│ (image and │     │ (inference  │     │ (structured│
│  targets)  │     │  and frame) │     │ prediction)│
└────────────┘     └─────────────┘     └──────┬──────┘
                                             │
                              ┌──────────────┴──────────────┐
                              ▼                             ▼
                       ┌───────────┐                 ┌────────────┐
                       │  Metrics  │                 │ Visualizers│
                       │ (scores)  │                 │ (optional) │
                       └───────────┘                 └────────────┘

The pipeline works as follows:

  1. DataLoader provides images together with ground-truth annotations.
  2. Engine runs inference and returns an EngineOutput; it also retains the corresponding frame for visualization.
  3. Parser converts the selected engine outputs into a structured prediction format.
  4. Metrics accumulate per-sample results and compute final scores.
  5. Visualizers independently consume the same predictions and optionally render them with the targets for inspection.

Because each component is resolved from a registry at runtime, you can mix and match implementations freely. For example, you can:

  • swap depthai for onnx in engine without changing the rest of the config
  • add another metric under pipeline.evaluators[*].metrics
  • introduce a custom parser and reference it by name
  • replace LuxonisLoader with a dataset-specific custom loader

The main constraint is compatibility: the parser must produce predictions in the format the configured metrics and visualizers expect, and the dataloader must provide the annotation keys those consumers require. LuxonisEval.setup() runs a one-sample pipeline sanity check across the loader, engine, parser, metrics, and active visualizers so incompatible configurations fail before evaluation starts. The check runs inference on one real sample, exercises each metric, and validates each visualizer's required target keys and prediction conversion.

📊 Throughput Metric Semantics

ThroughputMetric measures end-to-end pipeline timing rather than isolated model inference. The reported rows mean:

Warning

Throughput values cover the complete evaluation pipeline and do not represent isolated model inference performance. For model inference benchmarking, refer to modelconverter benchmark

  • Throughput - Samples processed per second across the full evaluation pipeline
  • End-to-end Latency - Average wall-clock time per sample for the whole run
  • Inference - Time spent inside the inference engine
  • Parsing - Time spent converting raw model outputs into predictions
  • Metric Update - Time spent updating metrics for each sample
  • Metric Compute - Time spent in the final metric aggregation after the sample loop
  • Pipeline Overhead - Remaining time not covered by the rows above; this typically includes dataloader iteration, image decode, preprocessing such as resize or normalization, annotation reconstruction, visualization, progress bar updates, and general loop bookkeeping

Rule of thumb: End-to-end Latency ≈ Inference + Parsing + Metric Update + Metric Compute + Pipeline Overhead

⚙️ Configuration

Evaluation runs are driven by a YAML configuration file. EvalConfig parses and validates the configuration at startup, ensuring that referenced components exist and that required fields are present before evaluation begins.

A typical configuration file only needs the pipeline block. version is filled from the package version, and runtime uses its default empty value when omitted.

pipeline:
  loader: ...
  engine: ...
  evaluators:
    - ...

📦 Data Loading And Preprocessing

This section defines which dataloader to use, which dataset it points to, and which preprocessing steps are applied before inference.

pipeline:
  loader:
    name: LuxonisLoader           # Registered dataloader name
    params:
      dataset_name: coco-2017     # Dataset identifier
      view: [val]                 # Dataset split(s) to use
    preprocessing:
      normalize:
        active: true              # Whether to apply normalization
        params:
          mean: [0.485, 0.456, 0.406]
          std: [0.229, 0.224, 0.225]
      color_space: RGB            # RGB | BGR | GRAY
      keep_aspect_ratio: true     # Preserve aspect ratio during resize

preprocessing is resolved before evaluation starts. For ordinary configs, the loader values come directly from YAML. When the model path points to an NNArchive, LuxonisEval can also derive preprocessing hints from the archive metadata.

runtime:
  nn_archive_params_override: true
pipeline:
  loader:
    preprocessing:
      keep_aspect_ratio: true
      normalize:
        active: false
      color_space: RGB

When runtime.nn_archive_params_override is true, NNArchive metadata takes precedence for:

  • loader.preprocessing.normalize
  • loader.preprocessing.color_space
  • evaluator parser params
  • evaluator outputs

Parser selection is handled separately: an explicitly configured parser name always wins, and the NNArchive parser is used only when the evaluator omits parser. When nn_archive_params_override is false, explicit YAML values stay primary for the fields above and archive metadata is only used as a fallback.

Important

keep_aspect_ratio is not inferred from NNArchive metadata. Set it explicitly when your preprocessing depends on preserving aspect ratio or letterboxing.

Note

For the depthai engine, host-side normalization is skipped because preprocessing is expected to run on-device through the NNArchive pipeline. For the onnx engine, resolved normalization stays on the host side.

Important

LuxonisLoader evaluation is currently single-evaluator and single-dataset-task only. pipeline.evaluators[*].task_name selects the Luxonis dataset task namespace to evaluate. It is not a framework-level task enum or abstraction. For datasets that use the default empty Luxonis task, set task_name: "".

🧠 Evaluators

Each pipeline evaluator binds together one dataset task selection, one parser, and its configured metrics and visualizers. At least one metric or one active visualizer is required, so metrics may be omitted for visualization-only runs.

pipeline:
  evaluators:
    - task_name: instance_segmentation
      parser:
        name: YOLOExtendedParser
        params:
          subtype: yolov8
          n_classes: 80
          conf_threshold: 0.25
          iou_threshold: 0.7
          mask_conf: 0.25
      metrics:
        - name: BboxMeanAveragePrecision
          params:
            iou_type: bbox
        - name: MaskMeanAveragePrecision
          params:
            iou_type: segm
      visualizers: []
  • task_name selects the Luxonis dataset task evaluated by this entry. If not set we try to infer it from the dataset metadata.
  • name is optional and defaults to task_name or a stable fallback when task_name is empty.
  • outputs is optional in the current single-evaluator implementation; when omitted, the evaluator consumes all engine outputs.
  • Exactly one evaluator is currently required at runtime. Omitted or multiple evaluators are rejected with a clear not-yet-implemented error.

Compatibility is driven by data shape, not by a separate task abstraction:

  • the loader must expose the annotation keys required by the configured metrics and visualizers
  • the parser must produce outputs that the configured metrics and visualizers can consume

🎨 Visualizers

Visualizers are configured per evaluator. BBoxVisualizer, InstanceSegmentationVisualizer, SegmentationVisualizer, and KeypointVisualizer render the ground truth and prediction side by side.

visualizers:
  - name: InstanceSegmentationVisualizer
    active: true
    display: false
    save: true
    save_dir: visualizations
    params:
      draw_labels: true
      draw_scores: true
      alpha: 0.6

At least one of display or save must be true. The configuration defaults are active: true, display: false, save: true, and save_dir: visualizations. In display mode, each image waits for a key press; pressing q or Escape closes the window and disables display for the rest of that evaluation.

Saved files use a task-specific prefix and a five-digit sequence number, such as bbox_00000.png. The sequence resets for every evaluate() call, so another evaluation using the same directory replaces files with matching names.

The built-in visualizers accept the following params, which match the corresponding LuxonisTrain visualizers. See the LuxonisTrain visualizer reference for detailed parameter descriptions and rendering examples.

Visualizer Required target keys Prediction type Main parameters
BBoxVisualizer [/boundingbox] dai.ImgDetections labels, draw_labels, draw_scores, colors, fill, width, font, font_size, scale
InstanceSegmentationVisualizer [/boundingbox, /instance_segmentation] dai.ImgDetections with instance-mask metadata Bounding-box parameters plus alpha and scale
KeypointVisualizer [/boundingbox, /keypoints] dai.ImgDetections with keypoints Bounding-box parameters plus visibility_threshold, connectivity, visible_color, nonvisible_color, radius, draw_indices
SegmentationVisualizer [/segmentation] dai.SegmentationMask colors, background_class, background_color, alpha, scale

All four renderers validate their prediction types and required target data during the setup sanity check. Parameter details and defaults are defined by the local implementations in luxonis_eval/visualizers.

⚡ Inference Engine

The engine section selects the inference engine and points to the model file. Configuration validation ensures that the model format matches the selected engine (.tar.xz NNArchive for depthai or onnx, .onnx for onnx).

pipeline:
  engine:
    name: onnx                    # Registered engine name: onnx | depthai
    model_path: ./models/yolov11n/yolov11n.onnx
    params: {}                    # Engine-specific parameters, for example device_ip for RVC4

Note

The CLI override flag is named --backend for convenience, but it simply overrides pipeline.engine.name.

📄 Full Example

runtime:
  nn_archive_params_override: true # Prefer preprocessing and parser parameters from the NNArchive.

pipeline:
  loader:
    name: LuxonisLoader
    preprocessing: # Sets preprocessing that is not represented by NNArchive metadata.
      # Letterbox images instead of stretching them to the model input size.
      # keep_aspect_ratio is not set by the NNArchive metadata and needs to be manually configured
      keep_aspect_ratio: true
    params:
      dataset_name: quickstartcoco
      view: [val]

  engine:
    name: onnx # Runs inference with ONNX Runtime on the host.
    model_path: examples/quickstart/yolov6.onnx.tar

  evaluators:
    - task_name: ""
      metrics:
        - name: BboxMeanAveragePrecision # Computes COCO-style bounding-box mean average precision.
          params:
            iou_type: bbox
      visualizers:
        - name: BBoxVisualizer
          display: false
          save: true # Writes rendered images to disk.
          save_dir: visualizations/quickstart/onnx

📏 Metrics

Quality metrics are configured per evaluator under pipeline.evaluators[*].metrics.

Metric Typical use Required target keys
TopKAccuracy Classification ["/classification"]
BboxMeanAveragePrecision Bounding box detection ["/boundingbox"]
MaskMeanAveragePrecision Instance segmentation ["/boundingbox", "/instance_segmentation"]
KeypointMeanAveragePrecision Keypoint evaluation ["/boundingbox", "/keypoints"]
MIoU Semantic segmentation ["/segmentation"]
DiceCoefficient Semantic segmentation ["/segmentation"]
F1Score Semantic segmentation ["/segmentation"]
JaccardIndex Semantic segmentation ["/segmentation"]

Metrics consume parser outputs directly. Each metric validates that the parser returned the message type it expects, for example classifications, segmentation masks, or detections. Instance-segmentation consumers derive per-instance masks from the indexed segmentation mask stored in dai.ImgDetections. Because this representation assigns each pixel to at most one instance, overlapping instance masks are not supported. To preserve overlapping masks, implement a custom parser and matching metric that exchange a different prediction message type.

ThroughputMetric is not configured manually in the evaluator list. It is always collected internally and reported alongside the quality metrics in the final EvaluationResult.

🏃 Commands

luxonis_eval eval --config ... runs the configured quality pipeline in this phase.

luxonis_eval quality --config ... is a quality-only alias with the same override flags as eval.

🧱 Extending the Framework

LuxonisEval is designed around a simple rule: implement a new class that inherits from the appropriate base class, and the registry handles the rest. Every component type (BaseEngine, BaseEvalLoader, BaseParser, BaseMetric, BaseVisualizer) uses AutoRegisterMeta, so subclassing is enough to make a component available once its module is imported.

📥 Adding a Custom DataLoader

Every custom loader must inherit from BaseEvalLoader and implement four abstract methods:

  • load_classes() - Returns a dict[str, int] mapping class names to integer indices. The result is assigned to self.classes and validated automatically.
  • get_class_mapping() - Returns a tuple of (ldf_class_map, native_class_map, class_index_map):
    • LDF class map (dict[int, str]): class ordering used inside Luxonis Data Format
    • Native class map (dict[int, str]): original class ordering used during training
    • Class index map (dict[int, int]): mapping from LDF indices to native indices
  • __getitem__(idx) - Returns a LoaderOutput tuple for the requested sample
  • __len__() - Returns the number of samples in the dataset

For LuxonisLoader-backed datasets, the LDF and native class maps may differ when the model was trained with a different class order than the dataset metadata, so the class index map must encode that remapping explicitly. If no native class mapping is provided for an unknown LuxonisLoader dataset, LuxonisEval falls back to the dataset's LDF class order and uses an identity class index map after issuing a warning. For custom datasets that inherit directly from BaseEvalLoader, the two class maps are usually identical and the class index map is typically an identity mapping.

Important

__getitem__ must return LoaderOutput from luxonis_ml.typing, which is a tuple of (image, annotations_dict).

  • image (np.ndarray) is a single image, for example with shape (H, W, 3).
  • annotations_dict (dict[str, np.ndarray]) maps task-group annotation keys to arrays, such as "/boundingbox", "/classification", or "/segmentation".
  • model_spec (ModelSpec) and nn_archive_cfg are passed to custom BaseEvalLoader constructors by LuxonisEval. Custom loaders can use the engine-resolved input/output tensor metadata, including model_spec.width and model_spec.height, during initialization or preprocessing setup. The built-in LuxonisLoader instead receives the resolved width and height directly.

Every subclass implementation of __getitem__ is wrapped by @validate_loader_output, which calls check_loader_output at runtime and raises a descriptive TypeError if the output format is invalid.

The loader must also provide a schema-stable annotations_dict: every sample must expose the same annotation keys. If a metric or active visualizer requires a key, that key must be present for every sample.

🔌 Adding a Custom Engine

Subclass BaseEngine, declare an output_type that subclasses EngineOutput, and implement the four abstract methods:

  • setup() - Initialize engine resources such as runtimes, sessions, or device connections, then return a ModelSpec(input=TensorSpec(...), outputs=(...)) for the loaded model. Its input must have a static 4D shape with NCHW or NHWC layout, and it must describe at least one output. Keep setup idempotent so repeated calls are safe.
  • infer_once(img) - Run inference on a single preprocessed image and return the engine's declared EngineOutput type
  • vis_frame() - Return the image associated with the latest inference in a form suitable for visualization overlays
  • close() - Release engine resources after evaluation finishes

The EngineOutput implementation exposes named tensors through names(), get(), and select(). The framework consumes the returned ModelSpec to configure loader preprocessing and builds an EvalContext that is attached to the parser, metrics, and visualizers.

🧠 Adding a Custom Parser

Subclass BaseParser and implement the single abstract method:

  • parse(output) - Convert an EngineOutput abstraction, potentially filtered by the evaluator's outputs setting, into a structured prediction format

Parser configuration belongs in the parser itself, and LuxonisEval provides the remaining runtime information during setup.

The parser bridges the gap between model-specific tensor layouts and the standardized message types that downstream metrics and visualizers expect. The built-in parsers produce the following output types:

Important

The parser must produce outputs that the configured metrics and visualizers can consume. For example, if a configured consumer expects dai.ImgDetections, the parser must return that message type.

📐 Adding a Custom Metric

Subclass BaseMetric and implement the four abstract methods:

  • required_target_keys() - Declare which annotation keys the metric requires
  • reset() - Reset internal state such as counters or accumulators
  • update(predictions, target) - Update the metric state for one sample
  • compute() - Return the final metric values

Important

Metrics must be compatible with the outputs generated by the configured parser. If the parser returns dai.ImgDetections, the metric must know how to process that object.

🪜 General Pattern

All extensions follow the same three-step workflow:

  1. Subclass the appropriate base class
  2. Implement the required abstract methods
  3. Reference the component by name in the YAML config

No manual registration, factory wiring, or extra boilerplate is required. As long as the module is imported, the metaclass makes the class available.

📄 License

This project is licensed under the Apache License 2.0.

About

A modular evaluation framework for benchmarking and analyzing deep learning models, designed to be easily extensible to new tasks and metrics, and to work seamlessly with the Luxonis ecosystem (DepthAI, Luxonis ML).

Resources

Readme
Apache-2.0 license
Activity
Custom properties

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages