Add Python objects: instances of Python classes owned by VHDL

[ru551n]

Adds Python objects to the bridge: instances of Python classes owned by VHDL, each living in a session of its own with state of its own. One can be a reference model, a scoreboard shared between a monitor and the test, a helper class in a testbench, or the backend of a verification component. That last use is the one that motivated the design. It replaces #14 and covers most of #8.

The problem

When a VC's behaviour lives in Python, every VC author has to solve the same problems:

• Namespaces. Functions and state in the default session collide when two instances of the same VC, or two models, use the same names. Sessions fix this, but each VC must create one with a unique identity, load its code into it, and pass it to every call.
• Locating the code. exec_file resolves relative to the testbench, which a VC deep in the hierarchy doesn't know. That leads to sys.path manipulation, passed-around paths, and ordering workarounds around test_runner_setup.
• String-built calls. Object and module names get concatenated into Python source, so mistakes only show up at run time.
• Sharing. There's no simple way for a testbench to create a model and give it to a VC, so that both use the same state.

The API

constant model : python_object_t := new_python_object("my_vcs.uart.UartModel", kwarg("baud", 115200), id => get_id("uart"));

count := call(model, "push", arg(byte));     -- method call, any result type of call()
call(model, "reset");                        -- procedure form
check_true(eval_boolean(model, "self.is_idle()"));
exec(model, "self.trace = True");

• The class is named, not loaded from a file. A dotted name like "package.module.Class" is imported from the module it names, found like any import: the run script directory, the Python environment, or PYTHONPATH. A bare "Class" is a class defined in the default session, and a name like "np.ndarray" follows a name defined there. A name that isn't found, or isn't a class, is reported in one line.

• One object, one session. The instance is bound to self in a session named by the object's identity, so objects never share names or state.

• Constructed on first use, or right away. new_python_object only records the class and the constructor arguments, so an object can be a constant or a generic, independent of test_runner_setup. create_python_object takes the same arguments and constructs the object in the call, so a wrong class or argument is reported there; it works only in a process, since NVC and GHDL can't run the bridge during elaboration. The user guide has a table of when to use each.

• Identity.

  • An object without an id is numbered like VUnit's VCs: python_bridge:python:object:1, :2, and so on.
  • An identity already used by another object, or by a session made with new_session, is reported on that identity's logger, and new_python_object returns null_python_object. Using null_python_object is reported on python_logger.
  • get_logger(object) reports the object's errors, named after what failed: call("push") failed: for every form of call, eval(...) and exec(...) for those, and new_python_object("<class>") failed:.

• VC backends. get_python_object(object, class_name, args, id) returns the given object or a new one, like get_logger/get_id. A VUnit-style VC calls it in its handle constructor, using the id from create_std_cfg. Each instance then gets its own backend, and a testbench can pass in, and keep using, its own:

  impure function new_uart_master(
    baud : positive; model : python_object_t := null_python_object; id : id_t := null_id
  ) return uart_master_t is
    constant std_cfg : std_cfg_t := create_std_cfg(id, "my_vcs", "uart_master");
  begin
    return (p_std_cfg => std_cfg,
            p_model => get_python_object(model, "my_vcs.uart.UartModel", kwarg("baud", baud), get_id(std_cfg)));
  end;

• Generated overloads. call and eval on an object come from the same result tables as python_pkg: every result type, the out-parameter procedures, and the same aliases. python_object_pkg.vhd is generated from hdl/tools/python_object_pkg.vhd.in by generate_python_pkg.py, and a pytest checks that both generated packages are up to date.

The identity comes from the handle rather than 'path_name, because GHDL drops the instance labels from 'path_name/'instance_name when they're evaluated during elaboration. A VC without a handle can use its instance path in a process body; the user guide shows both.

Changes

• hdl/tools/python_object_pkg.vhd.in and the generated hdl/src/python_object_pkg.vhd, added to python_context.
• runtime.py: __vunit__.instantiate_as_self. If the class can't be found or constructed, the error is reported once and self becomes a stand-in that raises the reason on every later use, assignments included. Classes are found with importlib; pydoc is not imported.
• On Riviera-PRO/Active-HDL, which have no sessions, new_python_object fails right away on the object's logger and returns null_python_object. The FFI packages tell the object package whether sessions are supported and which identities have sessions. The VHPI FFI gets a p_exec with an operation name, which only reports, and the result getters the generated object overloads use.
• Example: examples/embedded_python gets filter_vc, a clocked, handle-based VC whose behaviour is a MovingAverage object, with a test case driving two instances that have their own objects and one given the testbench's object.
• Docs:
  • a "Python modules" part of the "File paths" section from Report relative file names used before test_runner_setup and add base_path #12: the run script directory, the Python environment, PYTHONPATH, and global module names,
  • a "Python objects" section: the API, identity, a recipe for moving a model with module state to an object, VC backends, sharing, errors and limitations,
  • the fix from Document the design pattern for a verification component with a Python model #14 to the class example that used runner_cfg inside a component,
  • README and release note entries.

Tests

• tests/tb_python_objects.vhd covers:
  • handle-based VCs (tests/counter_vc_pkg.vhd, tests/counter_vc.vhd) with their own id, an enumerated id and a shared object,
  • a handle-less VC on its instance path (tests/path_counter_vc.vhd),
  • a class defined in the default session,
  • an identity already used by an object or a session,
  • an object that can't be created, reported by create_python_object right away and by the first use of a new_python_object, then on every use,
  • the naming of call errors, for the function and the procedure forms,
  • separate sessions and the numbering of anonymous objects,
  • every generated result type, and the eval procedures,
  • exec, get_logger, create_python_object and get_session, also before creation,
  • null_python_object,
  • objects with the identities of create_std_cfg, as in the user guide.
• pytest covers __vunit__.instantiate_as_self: dotted names including nested classes, names followed from the default session, default-session classes, a missing name, a name that isn't a class, import errors inside a model not being mistaken for a missing class, and the failed-creation stand-in.
• Locally:
  • the full suite (120 tests) and the example pass on NVC and GHDL,
  • pytest passes (120),
  • Sphinx builds without warnings,
  • the PYTHONPATH setup from the docs works with the run script started from another directory,
  • the VHPI package analyses cleanly with GHDL together with python_pkg and python_object_pkg, apart from its foreign bodies.

Coordination with #16 and #18

Two "Errors and limitations" bullets in the user guide describe behaviour those PRs change. Whichever PR merges second should drop the matching bullet:

• the bullet that string arguments are passed verbatim goes when Escape string arguments as Python string literals #16 merges,
• the bullet that integer_array_t constructor arguments can't be made during elaboration goes when Transfer integer_array_t arguments when they are first used #18 merges.

Question

• Bound name: the instance is self in its session. Is that the name we want?

🤖 Generated with Claude Code

[ru551n]

To claude: Why is this required?

[ru551n]

It wasn't required, so it's removed in 7306ba5. I'd added it only so that the Riviera-PRO/Active-HDL failure would say "Python objects require …". Their FFI already fails on the first use of a non-default session ("Python sessions are only supported with NVC, GHDL and Questa"), and every object uses a session of its own, so that message already covers it.

[ru551n]

Why is the colon required, cant we just have regular python namespace conventions?

[ru551n]

Also mentions that this calls the constructor

[ru551n]

Agreed, the colon is gone in 7306ba5. A class is now a plain dotted name, "package.module.Class". It's resolved by importing the longest prefix that is a module and taking the rest as attributes, so a nested class ("package.module.Outer.Inner") works without a separator. An import error inside the user's module is still reported as such, not mistaken for a missing module; pytest covers both cases.

[ru551n]

Done. The paragraph now says that new_python_object names the class and the arguments of its constructor, and that the constructor is called on first use (or right away by create).

[ru551n]

Is this required or implicit behaviour?

[ru551n]

Partly implicit. The run script's directory is the first entry of sys.path, and the packages of VUnit's Python environment (including pip install -e) are importable without anything extra. PYTHONPATH is only needed for code somewhere else, such as a VC's own Python folder. The new "Where Python code is found" chapter now spells this out.

[ru551n]

We should document the namespacing conventions (directories added to paths etc) in it's own chapter.

[ru551n]

Done: a new "Where Python code is found" chapter, before Sessions. It covers what the interpreter finds without setup (the run script directory, VUnit's Python environment), adding folders with PYTHONPATH from the run script, how exec_file/import_module_from_file resolve file names (and that exec_file temporarily adds the file's directory), and that sys.path/sys.modules are global to the simulation. The VC package layout moved there from the objects section, which now links to it.

[ru551n]

Rewrite this for better readability.

[ru551n]

Rewritten: it now leads with where a VC makes its object (the handle constructor, like VUnit's own components) and then what get_python_object does, before the example.

[ru551n]

This could also do with some readability improvements.

[ru551n]

Split into two shorter paragraphs. The first covers elaboration versus first call and the create_std_cfg numbering; the second points to filter_vc in the example and lists which file holds what.

[ru551n]

Readability.

[ru551n]

Reworded: "Several components can share an object, and so can a component and the testbench: the testbench makes the object and gives it to each of them." The scoreboard example follows. The closing sentence about a class-name generic is clearer too.

[ru551n]

This method should be named more of what it does.

[ru551n]

Renamed to __vunit__.instantiate_as_self, since it instantiates the named class and binds the instance to self in the current session.

[ru551n]

This is not very clear if the model is found automatically or not, since I think these snippets are in the wrong order.

[ru551n]

Agreed, the order hid it. Reworked in f10700d. The chapter now has a subsection per topic: "Found without setup", "Adding a folder", "Files given by name" and "Module names". "Adding a folder" opens with "Python code anywhere else is not found until its folder is on PYTHONPATH". It then shows the VC layout, says outright that my_vc/python can't be imported as it is, shows the run script snippet, and only then says that my_vc_model is importable everywhere. pip install -e is mentioned there as the alternative.