tools/vtk_test_ports: VTK's Python tests with Tack's algorithms in place of VTK's filters

tools/vtk_test_ports/README.md

@@ -0,0 +1,57 @@
+# VTK's Python tests on Tack
+
+VTK's own Python tests, run with Tack's VTK algorithms (`tack.interop.vtk.algorithms`)
+in place of VTK's filters: a robustness test of Tack's filters on the data and
+pipelines VTK's tests build. It is not a check that Tack replaces VTK's filters one
+for one -- Tack keeps its own array names, cell orders and conventions.
+
+## How it works
+
+`shims.py` puts a stand-in in place of each VTK class it knows (`vtkContourFilter`,
+`vtkCutter`, `vtkClipDataSet`, `vtkStreamTracer`, ...; `TRANSLATIONS`), in its
+`vtkmodules` module, so a test script runs unchanged. A stand-in keeps a real VTK
+filter as its twin: every call the stand-in does not answer (`SetValue`,
+`SetCutFunction`, ...) goes to the twin, so the twin holds the test's settings. When
+the pipeline runs the stand-in, it reads the twin's settings, runs the Tack wrapper
+they translate to, and gives its output (converted to the class VTK's filter gives).
+It also runs the twin on the same input and records the two outputs compared:
+counts, bounds over the points cells use, total length, area and volume, and each
+common array's range, with a verdict (`same`, `same geometry` where only counts
+differ, `differ`). A setting with no Tack counterpart raises `Unsupported`, and a
+Tack error is caught: the stand-in then gives the twin's output, so the test still
+runs, and the record says why.
+
+`run_one.py` runs one test as VTK's test driver does (`rtImageTest`), rendering
+offscreen, with VTK's filters (`vtk`) or Tack's (`tack`), its image checked against
+VTK's baseline. `drive.py` runs a list of tests both ways in parallel into
+`summary.json`; `report.py` summarizes it: each test's outcome, per VTK filter how
+Tack's output compared, the errors, the unsupported settings, the outputs that
+differ, and the active attributes VTK sets.
+
+## Running it
+
+It needs a VTK source tree (for the tests), VTK's test data and baselines (a VTK
+build's `ExternalData`), and a VTK with rendering for Python (a local build on
+`PYTHONPATH`, or the wheel):
+
+```sh
+uv sync --extra cpu --extra dev --extra interop
+grep -rl "vtkContourFilter" ~/VTK/Filters/*/Testing/Python/*.py \
+    | sed "s#$HOME/VTK/##" > tests.txt                # the tests to run
+uv run python tools/vtk_test_ports/drive.py ~/VTK ~/VTK-build/ExternalData out tests.txt 4
+uv run python tools/vtk_test_ports/report.py out/summary.json
+```
+
+Each test's output, its image and diff, and the stand-ins' `records.json` are in
+`out/<module>_<test>/{vtk,tack}/`.
+
+## What it found (2026-11-05)
+
+Of VTK's 444 Python tests that use a filter Tack has, 239 pass with Tack's filters
+in place after the fixes it led to: higher-order field writes (#343), triangulation
+of concave polygons (#344), transformed vectors and normals (#345), active
+attributes and tube normals (#348), vector components (#349), serendipity cells
+with 64-bit data and flat curvilinear extents (#350), hidden points, connectivity
+on polydata and empty polygons (#352), polydata outputs (#353). The rest are
+expected differences (integrators, triangulations, exact higher order), VTK's names
+and cell orders, or settings the stand-ins do not translate.

tools/vtk_test_ports/drive.py

@@ -0,0 +1,74 @@
+# SPDX-FileCopyrightText: Copyright (c) Kitware Inc.
+# SPDX-License-Identifier: BSD-3-Clause
+
+"""Run VTK's Python tests with VTK's filters and with Tack's (``run_one.py``), in
+parallel, and gather what each gave into ``summary.json``:
+
+    python drive.py <VTK source> <VTK ExternalData> <output directory> <tests file> [workers]
+
+``tests file`` lists test scripts, one per line, relative to the VTK source."""
+
+import concurrent.futures
+import json
+import os
+import re
+import subprocess
+import sys
+import time
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+
+
+def run(source, data, out, test, mode):
+    module = test.split("/Testing/Python/")[0]
+    name = os.path.splitext(os.path.basename(test))[0]
+    baseline = os.path.join(data, module, "Testing/Data/Baseline", name + ".png")
+    where = os.path.join(out, module.replace("/", "_") + "_" + name, mode)
+    command = [sys.executable, os.path.join(HERE, "run_one.py"), os.path.join(source, test),
+               mode, where, os.path.join(data, "Testing")]
+    if os.path.exists(baseline):
+        command.append(baseline)
+    start = time.time()
+    try:
+        done = subprocess.run(command, capture_output=True, text=True, timeout=600,
+                              cwd=os.path.dirname(os.path.join(source, test)))
+        output, code = done.stdout + done.stderr, done.returncode
+    except subprocess.TimeoutExpired as expired:
+        output, code = (expired.stdout or b"").decode(errors="replace") if isinstance(
+            expired.stdout, bytes) else (expired.stdout or ""), "timeout"
+    error = re.findall(r'name="ImageError" type="numeric/double">([^<]+)<', output)
+    passed = "RESULT PASSED" in output
+    traceback = ""
+    if not passed:
+        tb = output.rfind("Traceback (most recent call last)")
+        traceback = output[tb:tb + 3000] if tb >= 0 else output[-1500:]
+    records = []
+    path = os.path.join(where, "records.json")
+    if mode == "tack" and os.path.exists(path):
+        with open(path) as f:
+            records = json.load(f)["records"]
+    return {"test": test, "mode": mode, "passed": passed, "code": code,
+            "image_error": [float(e) for e in error], "baseline": os.path.exists(baseline),
+            "seconds": round(time.time() - start, 1), "records": records,
+            "traceback": traceback}
+
+
+def main():
+    source, data, out, listing = sys.argv[1:5]
+    workers = int(sys.argv[5]) if len(sys.argv) > 5 else 4
+    tests = [line.strip() for line in open(listing) if line.strip()]
+    os.makedirs(out, exist_ok=True)
+    results = []
+    with concurrent.futures.ThreadPoolExecutor(workers) as pool:
+        futures = [pool.submit(run, source, data, out, t, m) for t in tests for m in ("vtk", "tack")]
+        for i, future in enumerate(concurrent.futures.as_completed(futures)):
+            r = future.result()
+            results.append(r)
+            print(f"[{i + 1}/{len(futures)}] {r['mode']:4} {'ok  ' if r['passed'] else 'FAIL'} "
+                  f"{r['test']} {r['image_error']} {r['seconds']}s", flush=True)
+            with open(os.path.join(out, "summary.json"), "w") as f:
+                json.dump(results, f, indent=1)
+
+
+if __name__ == "__main__":
+    main()

tools/vtk_test_ports/report.py

@@ -0,0 +1,80 @@
+# SPDX-FileCopyrightText: Copyright (c) Kitware Inc.
+# SPDX-License-Identifier: BSD-3-Clause
+
+"""What ``drive.py``'s ``summary.json`` says, as text: each test's outcome, and per
+VTK filter how Tack's output compared, what was not supported and what failed.
+
+    python report.py <summary.json>"""
+
+import collections
+import json
+import sys
+
+
+def outcome(vtk, tack):
+    used = [r for r in tack["records"] if r.get("used") == "tack"]
+    if not vtk["passed"]:
+        return "VTK fails here"
+    if not tack["records"]:
+        return "no stand-in ran" if tack["passed"] else "fails before a stand-in ran"
+    if not used:
+        return "VTK's output used throughout"
+    if tack["passed"]:
+        return "Tack used, test passes"
+    return "Tack used, test fails"
+
+
+def main():
+    results = json.load(open(sys.argv[1]))
+    by_test = collections.defaultdict(dict)
+    for r in results:
+        by_test[r["test"]][r["mode"]] = r
+    outcomes = collections.defaultdict(list)
+    for test, modes in sorted(by_test.items()):
+        if "vtk" in modes and "tack" in modes:
+            outcomes[outcome(modes["vtk"], modes["tack"])].append(test)
+    print("== Tests")
+    for name, tests in outcomes.items():
+        print(f"{len(tests):4}  {name}")
+    print()
+    print("== Tack used, test fails")
+    for test in outcomes["Tack used, test fails"]:
+        tack = by_test[test]["tack"]
+        verdicts = collections.Counter(r.get("verdict") or r.get("used") for r in tack["records"])
+        print(f"  {test}  image {tack['image_error']}  {dict(verdicts)}")
+        if not tack["image_error"]:
+            print("    " + tack["traceback"][-400:].replace("\n", "\n    "))
+    records = [rec for modes in by_test.values() if "tack" in modes for rec in modes["tack"]["records"]]
+    print()
+    print("== Per VTK filter: verdicts of Tack's output against VTK's")
+    per = collections.defaultdict(collections.Counter)
+    for rec in records:
+        per[rec["filter"]][rec.get("verdict") or ("unsupported" if "why" in rec else "error")] += 1
+    for name, counts in sorted(per.items(), key=lambda kv: -sum(kv[1].values())):
+        print(f"  {name:28} " + ", ".join(f"{k} {v}" for k, v in counts.most_common()))
+    print()
+    print("== Errors (Tack raised), by message")
+    errors = collections.Counter((rec["filter"], rec["error"][:160]) for rec in records if "error" in rec)
+    for (name, message), n in errors.most_common():
+        print(f"  {n:3}  {name}: {message}")
+    print()
+    print("== Unsupported settings")
+    why = collections.Counter((rec["filter"], rec["why"]) for rec in records if "why" in rec)
+    for (name, message), n in why.most_common():
+        print(f"  {n:3}  {name}: {message}")
+    print()
+    print("== Outputs that differ")
+    for rec in records:
+        if rec.get("verdict") == "differ":
+            print(f"  {rec['filter']} ({rec['test'].split('/')[-1]}): points {rec['points']} "
+                  f"cells {rec['cells']}: " + "; ".join(rec["notes"])[:400])
+    print()
+    print("== Active attributes VTK sets and Tack does not")
+    active = collections.Counter((rec["filter"], note.split(" active ")[1].split(" in ")[0])
+                                 for rec in records for note in rec.get("active", []))
+    for (name, what), n in active.most_common():
+        print(f"  {n:3}  {name}: {what}")
+
+
+if __name__ == "__main__":
+    main()

tools/vtk_test_ports/run_one.py

@@ -0,0 +1,108 @@
+# SPDX-FileCopyrightText: Copyright (c) Kitware Inc.
+# SPDX-License-Identifier: BSD-3-Clause
+
+"""Run one of VTK's Python tests as VTK's test driver does (``rtImageTest``), with
+VTK's filters (``vtk``) or with Tack's in their place (``tack``, ``shims.py``):
+
+    python run_one.py <test.py> vtk|tack <output directory> <data root> [<baseline png>]
+
+Rendering is offscreen. The test's image goes to the output directory, compared
+with the baseline by vtkTesting (its ``ImageError`` measurement on stdout); with
+``tack``, the stand-ins' records go to ``records.json`` there."""
+
+import atexit
+import json
+import os
+import sys
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+
+
+def offscreen():
+    """Every render window offscreen: ``vtkRenderWindow`` (in its module, and in
+    ``vtkmodules.all`` and ``vtk`` when imported) made one that renders offscreen.
+    VTK overrides a class only with a pure Python subclass, so the name is wrapped."""
+    import vtkmodules.vtkRenderingCore as core
+    import vtkmodules.vtkRenderingOpenGL2  # noqa: F401 -- the render window factory
+
+    real = core.vtkRenderWindow
+
+    def vtkRenderWindow(*args, **kwargs):
+        window = real(*args, **kwargs)
+        window.SetOffScreenRendering(1)
+        return window
+
+    core.vtkRenderWindow = vtkRenderWindow
+    import vtkmodules.all as everything
+
+    everything.vtkRenderWindow = vtkRenderWindow
+
+
+def main():
+    test, mode, out, data = sys.argv[1:5]
+    baseline = sys.argv[5] if len(sys.argv) > 5 else None
+    os.makedirs(out, exist_ok=True)
+    offscreen()
+    if mode == "tack":
+        sys.path.insert(0, HERE)
+        import shims
+
+        import tack
+
+        tack.set_default_device("cpu")
+        replaced = shims.install()
+        shims.CURRENT["test"] = test
+
+        def write():
+            with open(os.path.join(out, "records.json"), "w") as f:
+                json.dump({"replaced": replaced, "records": shims.LOG}, f, indent=1,
+                          default=lambda o: o.tolist() if hasattr(o, "tolist") else str(o))
+
+        atexit.register(write)
+    from vtkmodules.test import rtImageTest
+    from vtkmodules.vtkTestingRendering import vtkTesting
+
+    args = [test, "-D", data, "-T", out]
+    if baseline:
+        args += ["-V", baseline, "-B", os.path.dirname(baseline)]
+    sys.argv = ["rtImageTest.py", *args]
+    try:
+        code = rtImageTest.main(test) if baseline else run_without_image(test)
+    except SystemExit as exit:                      # unittest-style tests (Testing.main)
+        code = vtkTesting.PASSED if exit.code in (0, None) else vtkTesting.FAILED
+    print(f"RESULT {'PASSED' if code == vtkTesting.PASSED else 'FAILED'}", flush=True)
+    sys.exit(0 if code == vtkTesting.PASSED else 1)
+
+
+def run_without_image(test):
+    """A test with no baseline image, as CTest runs it: passed if it runs to the end
+    (rtImageTest calls a test without an image failed). The interactor's Start does
+    nothing, as rtImageTest makes it."""
+    import traceback
+
+    import vtkmodules.vtkRenderingUI  # noqa: F401 -- the interactor factory
+    from vtkmodules.vtkRenderingCore import vtkRenderWindowInteractor
+    from vtkmodules.vtkTestingRendering import vtkTesting
+
+    kind = type(vtkRenderWindowInteractor())
+
+    class Interactor(kind):
+        def Start(self):
+            pass
+
+    kind.override(Interactor)
+    directory = os.path.abspath(os.path.dirname(test))
+    sys.path.insert(0, directory)
+    try:
+        with open(test) as f:
+            exec(compile(f.read(), test, "exec"), {"__name__": "__main__"})
+    except SystemExit:
+        raise
+    except Exception:
+        traceback.print_exc()
+        return vtkTesting.FAILED
+    return vtkTesting.PASSED
+
+
+if __name__ == "__main__":
+    main()