torx

torx is a distributed testing and benchmarking framework. It is inspired by Ceph's teuthology and Confluent's ducktape. Both of those frameworks are written in Python, but the software they test can be written in any language: Ceph and Redpanda are C++ systems tested with them, and Kafka is a Java one.

Like those frameworks, torx models a test as a job in a suite. A job declares the services it needs and asserts on their behavior, and the machines under it sit behind abstractions such as Node and Backend, so that many kinds of networked software can be tested the same way.

The suite binary is the driver. It discovers the jobs compiled into it, works out how many nodes each one needs from the services it declares, allocates those nodes from the pool, and runs each job in a worker, which is the same binary re-executed. Inside the worker the job's services start their processes on the nodes they were given, and every command, file transfer, and signal a service issues goes through the node's backend. torx ships two backends, local subprocesses and ssh. The tutorial's docker nodes are containers running sshd, reached through the ssh backend; a backend that drove containers through docker exec would register the same way and change nothing above it.

Getting started

The examples are the way in. Each has a README that says how to run it and what to look at.

• examples/tutorial/02-service is the smallest complete suite: a service that runs one server on one node, and a job that writes a key to it and reads the key back. It is the second step of the tutorial, six suites that each build on the one before.
• examples/tutorial/06-launcher wraps the suite in a launcher. torx does not build the suite, install the system under test, or provision nodes, so in practice a project runs its suite through a harness that does. This one builds the binaries, records how the run was invoked, provisions docker containers, and runs the suite against them over ssh.
• examples/rqlite tests a real distributed database, rqlite, with the same building blocks: a cluster service whose nodes join in order, parametrized and fault-injection jobs, a backup taken and restored, and the launcher a team would run it through.

Reference

A suite is a Go binary that links the torx library and its own jobs; the same binary is both the driver and, re-executed, the worker that runs one job. You write two kinds of thing:

• a Service, which says how to run a process (a server, a client, a load generator) on the nodes allocated to it, and
• a Job, which is one test or benchmark: it declares the services it needs and drives them.

The framework does the rest: it sizes each job from the services it declares, allocates a disjoint set of nodes for it, starts the services, waits for readiness, runs the job body, and tears everything down, collecting logs and artifacts along the way.

A few concepts you will meet:

• Node is one execution target. A node runs commands and moves files through its Backend (LocalBackend for local runs, the ssh backend for remote nodes); a service never touches the transport, it calls node.Exec, node.Stream, node.WriteFile, and so on. A node also carries a scratch directory, a port allocator, and its reachable address (node.Addr()).
• Pool is the finite set of nodes a run owns. The driver allocates a sub-pool per job and frees it on completion, so two jobs never share a node.
• Result is what a job leaves behind. A job passes when Run returns nil. A benchmark additionally records an opaque Data payload plus a one-line Summary; torx stores these verbatim and never interprets them.

Authoring a Service

A service embeds *torx.ServiceBase and implements the four per-node lifecycle hooks. ServiceBase turns those hooks into the coarse Start/Stop/Clean/ Wait lifecycle the framework drives, stopping and cleaning each node before starting it so a service always begins from a known state.

package myservice

import (
	"context"
	"errors"
	"net"
	"strconv"
	"sync"
	"syscall"
	"time"

	"github.com/dotnwat/torx"
)

// Service runs one myserver per node.
type Service struct {
	*torx.ServiceBase

	mu      sync.Mutex
	servers map[string]torx.Process // node name -> running server
	addrs   map[string]string       // node name -> host:port
}

// New builds a service named name that needs one node.
func New(name string) *Service {
	s := &Service{servers: map[string]torx.Process{}, addrs: map[string]string{}}
	// Homogeneous(count, spec) is the node demand. A spec can require CPUs,
	// memory, or labels (torx.NodeSpec{Required: torx.Resources{...}}); an empty
	// spec matches any node.
	s.ServiceBase = torx.NewServiceBase(name, torx.Homogeneous(1, torx.NodeSpec{}), s)
	return s
}

// StartNode launches the server on n. It leases a free port, binds every
// interface so a client off the node can reach it, and advertises the node's
// reachable address. StartCaptured runs the process with its output redirected
// to a node-local file, registers that file for collection, and returns the
// process to signal, wait for, and stop.
func (s *Service) StartNode(ctx context.Context, n *torx.Node) error {
	port, err := n.AllocatePort()
	if err != nil {
		return err
	}
	dir := n.ServiceScratch(s.Name()).Root // a disjoint scratch dir for this service
	cmd := torx.Command("myserver",
		"--listen", "0.0.0.0",
		"--port", strconv.Itoa(port),
		"--dir", dir,
	)
	p, err := s.StartCaptured(ctx, n, cmd)
	if err != nil {
		return err
	}
	s.mu.Lock()
	s.servers[n.Name()] = p
	s.addrs[n.Name()] = net.JoinHostPort(n.Addr(), strconv.Itoa(port))
	s.mu.Unlock()
	return nil
}

// WaitNode blocks until the server is ready. Readiness is a real check, never a
// bare sleep: poll a port, an HTTP endpoint, or a protocol ping.
func (s *Service) WaitNode(ctx context.Context, n *torx.Node) error {
	s.mu.Lock()
	addr := s.addrs[n.Name()]
	s.mu.Unlock()
	return torx.WaitForPort(ctx, addr) // or torx.WaitUntil(ctx, poll, backoff)
}

// StopNode stops the server the way an operator would: SIGTERM, a grace period
// to exit on its own, and a SIGKILL of its whole process group if it has not.
// A server that had to be killed is noted rather than failed. The node is
// clean either way, and a teardown error would quarantine it.
func (s *Service) StopNode(ctx context.Context, n *torx.Node) error {
	s.mu.Lock()
	p := s.servers[n.Name()]
	delete(s.servers, n.Name())
	delete(s.addrs, n.Name())
	s.mu.Unlock()
	if p == nil {
		return nil
	}
	_, err := torx.Shutdown(ctx, p, syscall.SIGTERM, 5*time.Second)
	if errors.Is(err, torx.ErrShutdownTimeout) {
		torx.Logf(ctx, "warn", "myserver on %s ignored SIGTERM and was killed", n.Name())
		return nil
	}
	return err
}

// CleanNode removes the server's persistent state.
func (s *Service) CleanNode(ctx context.Context, n *torx.Node) error {
	return n.Rm(ctx, n.ServiceScratch(s.Name()).Root)
}

// Addr exposes the server's advertised address to jobs and other services.
func (s *Service) Addr() string {
	s.mu.Lock()
	defer s.mu.Unlock()
	for _, n := range s.Nodes() {
		if a, ok := s.addrs[n.Name()]; ok {
			return a
		}
	}
	return ""
}

Guidelines that keep a service portable across the local and ssh backends:

• Reach the node only through its methods: n.Exec, n.Stream, n.ReadFile, n.WriteFile, n.Mkdir, n.Rm, n.Signal. They run on the node whether that is a local subprocess or a container over SSH.
• Bind broadly, advertise n.Addr(). Bind the server to all interfaces so a client on another node can reach it, and record n.Addr() (not a hardcoded 127.0.0.1) as the address others dial.
• Lease ports with n.AllocatePort() rather than hardcoding one, so services co-located on a host do not collide.
• Capture output with StartCaptured. It redirects the process's stdout and stderr to a node-local file and collects it into the results tree. For any other output (a --log-file, a data dump) call s.AddArtifact(n, torx.Artifact{Name: ..., Path: ..., CollectOnPass: true}). A service that launches more than one process on a node over a job, as a crash-and-restart test does, calls s.SetCapturePolicy(torx.CaptureRotate) at construction so each launch moves the previous process's log aside as stdout.<k>.log and collects it too, instead of discarding it (the default, CaptureTruncate). A service that runs several processes on one node at once -- a database's writer beside its compactor -- gives each a log of its own with s.StartCapturedAs(ctx, n, name, cmd), which writes <name>.log and rotates each name's launches apart; s.CapturePath(n, name) says where. A name may not end in a dot and a number, as a rotated log's does.
• Stop through the process handle. StartCaptured (and n.Stream) return a torx.Process: Signal reaches the program itself, not a shell around it, Wait reports its exit status, and Close kills its whole process group. torx.Shutdown(ctx, p, syscall.SIGTERM, grace) composes them into the usual graceful stop and escalates to the kill when the grace period runs out (reported as torx.ErrShutdownTimeout). Use it so teardown goes through the system's own shutdown path, and to drive rolling restarts and signal-triggered reloads from a job. torx.Stop(ctx, p, torx.StopPolicy{...}) is the same stop with one more step: a Dump signal sent when the grace period runs out, before the kill, on which the program writes out where it is stuck. syscall.SIGQUIT has a Go program print every goroutine's stack (and a JVM a thread dump) into the log StartCaptured collects, so a stop that hangs leaves the evidence of why. Signal also pauses and resumes a process (SIGSTOP, SIGCONT), the fault that makes a node go silent without dying.
• One-shot commands (a load generator that runs and exits) use n.Exec(ctx, cmd) instead of StartCaptured; it runs to completion and returns the captured ExecResult (exit code, stdout, stderr).

A service that is not a per-node server, such as a rolling restart, a one-shot client, or a single cloud-API call, can override the coarse lifecycle methods (Start, Stop, Clean, Wait) directly instead of implementing the per-node hooks; NewServiceBase then takes nil for the hooks.

Authoring a Job (test or benchmark)

A job embeds torx.JobBase and implements Declare and Run.

package main

import (
	"context"

	"example.com/myservice"
	"github.com/dotnwat/torx"
)

func main() { torx.Main() }

// Register the job by a stable id in an init function so the driver can discover
// it and the worker can reconstruct it.
func init() {
	torx.Register("my.smoke", func() torx.Job { return &smokeJob{} })
}

type smokeJob struct {
	torx.JobBase
	server *myservice.Service
}

// Declare registers and configures the services the job needs. It must be pure:
// construct and register services, but do not allocate or start anything. The
// framework calls it to size the job and the worker calls it to rebuild the job
// identically.
func (j *smokeJob) Declare(jc *torx.JobContext) {
	j.server = myservice.New("myserver")
	jc.Register(j.server)
}

// Run is the test body. It runs after the framework has started every declared
// service and waited for readiness. Return nil to pass, an error to fail.
func (j *smokeJob) Run(ctx context.Context, jc *torx.JobContext) error {
	addr := j.server.Addr()
	// ... connect to addr, exercise the server, assert behavior ...
	jc.SetSummary("myserver came up and answered")
	return nil
}

What JobBase gives you, and how to take control:

• Setup starts every declared service in registration order, then waits for each to be ready. Override Setup to control start order or start lazily.
• Teardown stops the services, collects their artifacts, cleans them, and runs finalizers, in that order and each in reverse registration order, aggregating every error. Override it only if you need to, and call jc.CollectArtifacts(ctx) between stopping and cleaning so logs survive.
• jc.Defer(fn) registers a cleanup callback run during teardown.
• jc.WriteArtifact(name, data) keeps a file the job itself produced, such as a backup it took, a report it downloaded, or a histogram it measured, at the top of its results directory, beside result.json; the services' collected files sit in directories below. The name must be a single path component that is not a framework file or a service name in any letter case (a bad one panics), writing it again replaces the file, concurrent writes are serialized, and the file is kept whatever the outcome. In a run that is not persisting results it is a no-op. A write that fails is recorded on the result as a persistence failure, which makes the suite not Ok, rather than failing the job over a full disk.
• A benchmark records what it measured: jc.Record(anyValue) stores an opaque JSON payload and jc.SetSummary("...") a one-line human summary. torx never interprets Data; large outputs belong in artifacts, written with jc.WriteArtifact when the job holds them and collected from nodes when a service produced them.
• A randomized job (a chaos test, a random-operations test, a configuration drawn from a space of options) draws from jc.Rand("name"), a generator for one named stream of the variant's seed, and gives each goroutine a stream of its own. A run has one seed, drawn at random unless -seed N fixes it, printed with the run's summary and recorded in run.json; each variant's seed is derived from it and the variant's id (torx.VariantSeed) and recorded on its result. Rerunning with the same -seed and a selection of just the failing variant hands it the same seed, so it makes the same choices again. The seed is the same in Declare as in the worker, so a job may even draw its shape from it. A matrix of a trial parameter runs one job under many seeds.

Multi-service jobs just declare more services; the framework sums their demand into the pool it allocates and hands each service its nodes (Bind) in registration order. A job accesses one service from another through the service's own methods (e.g. server.Addr()), exactly as in Run above.

Parametrization. A job may expand into several variants by implementing Matrix() []torx.Params; the torx.Matrix(map[string][]any) helper builds the cross product. Each variant gets a stable id and is selected, scheduled, and reported independently. Read jc.Params (with the typed Int/String/Bool getters) inside Declare/Run.

A parametrized job, and above all one meant to take externally supplied configurations (-params, below), should also implement

ResolveParams(p torx.Params) (torx.Params, error)

Discovery calls it once per variant, before selection, duplicate detection, and id construction. The job returns the complete canonical map: defaults filled in, values type- and range-checked, unknown keys rejected. That map is what the variant runs with, what its id is computed from, and what its results record. Without the hook, {a:1} and {a:1, b:<default>} are two different ids for the same configuration, and a mistyped key silently runs the default value. An error from the resolver fails the variant loudly before any node is allocated. One consequence worth knowing: adding a dimension later changes every canonical id (its default joins every map), so join runs on the recorded params in results, not on id strings.

Wiring it into a suite binary

A suite is a Go binary whose main calls torx.Main() and whose jobs are registered via init. A job is discoverable only if its package is linked into that binary, so the file with the torx.Register(...) call must be imported (the main package here contains it directly).

Running a suite

The binary is the driver by default. Positional arguments select jobs by id (regular expressions); with none, every registered job runs.

# Local pool sized to the largest job (or fix it with -nodes N):
go run ./path/to/suite my.smoke
go run ./path/to/suite -nodes 3 'my\..*'

# Results land under ./results/<timestamp>/ with a `latest` symlink:
#   results/<ts>/run.json
#   results/<ts>/<jobVariant>/{events.ndjson, test_log, result.json,
#                              <the job's own artifacts>,
#                              <service>/<node>/stdout.log[, stdout.<k>.log]}

Useful flags: -nodes N (local pool size), -parallel N (concurrent jobs), -results <file> (newline-delimited JSON results), -results-dir <dir> (the per-run tree; empty to disable), -run-dir <dir> (below), -params <file> (below), -pool <manifest.json> (below), -netns (local nodes with networks of their own; below), -cgroups (local nodes with cgroups of their own; below), and -seed N (the run seed a randomized job draws from; above).

External parametrization. -params FILE replaces the named jobs' compiled-in variants with externally supplied ones, so a specific configuration or sweep runs without editing the suite. The file is JSON keyed by job id; each entry gives matrix (a dimension name to its list of values, expanded to the cross product), configs (explicit parameter objects, for curated points a cross product cannot express), or both (the expanded matrix plus the configs):

{
  "my.bench": {
    "matrix":  { "clients": [1, 8, 32], "trial": [1, 2, 3] },
    "configs": [ { "clients": 64, "pipeline": 8 } ]
  }
}

Entries replace a job's compiled-in variants entirely; nothing is merged. The envelope is strict, because externally supplied configuration must never degrade silently: unknown fields, empty forms, empty dimensions, and null values are rejected, an entry expanding to more than torx.MaxVariants (65536) variants is refused with the count named, and naming a job that is unknown or whose variants end up entirely unselected is an error. Each supplied parameter set still passes through the job's ResolveParams (above), so ids stay canonical and two entries that resolve to the same configuration are rejected as duplicates.

Launchers and -run-dir. A tool that wraps a suite (building it, writing an invocation record, archiving the run's inputs) needs to know the exact run directory before the run starts; the tutorial's launcher and examples/rqlite/harness are two such tools. Resolving the latest symlink afterwards races concurrent runs, and an interrupted run would leave the metadata with no home at all. Such a launcher creates the run directory itself, writes its metadata into it, and then invokes the suite with -run-dir DIR. torx uses the directory exactly as given: no timestamped subdirectory is minted and no latest symlink is maintained, since those conveniences belong to -results-dir mode, and torx fills in the per-variant subdirectories and the final run.json. The directory must already exist, and -run-dir is mutually exclusive with -results-dir. A launcher that wants its runs named and linked the way -results-dir mode does it mints the directory with torx.MakeRunDir(root), the same call the driver makes: a UTC timestamp plus a random suffix, unique under same-second starts, with latest swapped in atomically.

Remote pools. To run against nodes a provisioner prepared, pass a node manifest with -pool and blank-import the ssh backend in your suite's main so ssh-kind nodes are constructible:

import _ "github.com/dotnwat/torx/ssh"

{
  "nodes": [
    { "name": "n0", "address": "10.0.0.5", "scratch": "/var/tmp/torx",
      "ports": { "min": 30000, "max": 31000 },
      "backend": { "kind": "ssh", "host": "10.0.0.5",
        "config": { "user": "torx", "identity_file": "/etc/torx/id_ed25519",
                    "known_hosts": "/etc/torx/known_hosts" } } }
  ]
}

A node needs a POSIX sh. To kill a service's whole process group on teardown, and to signal the service's own process rather than a shell around it, torx runs each streamed command as the leader of its own process group. Under OpenSSH with a bash or zsh login shell the command already is one, and nothing else is needed; macOS Remote Login is such a setup. Otherwise the wrapper creates the group with setsid from util-linux or busybox, or with perl, whichever the node has. That covers a login shell that forks (dash), ForceCommand wrappers, and an sshd that does not isolate commands at all (Dropbear). A node with none of those refuses to stream rather than risk signalling the sshd itself, and the error says what to install.

Local nodes with networks of their own (-netns). The local pool's nodes share the host's network, so a job cannot cut one off from another: there is no link between them to cut. On Linux, -netns gives each local node a network namespace of its own, joined to the others by a bridge, with an address of its own (10.77.0.2, 10.77.0.3, ...), which is what node.Addr() returns. No root is needed: the driver re-executes itself in a user namespace of its own, where it has privilege over the namespaces it creates and nothing else, and the driver and its workers live on the bridge, so they reach every node while nothing in the lab reaches the host's network. A node's commands enter its namespace through nsenter, and its files are the host's, as for any local node. Each node's ports are its own, so a service that leases ports works unchanged. It needs ip, nsenter, and sleep on the host and a kernel that lets an unprivileged user create a user namespace (Ubuntu restricts that by default; see kernel.apparmor_restrict_unprivileged_userns).

go run ./path/to/suite -netns -nodes 6 -parallel 2 'my\.chaos'

The netfault package injects faults into that network: netfault.Partition(ctx, groups...) (overlapping groups make a bridge), Isolate, Block (one way), Blackhole (drop only large packets, an MTU black hole), Heal, and SetShape/Unshape for delay, jitter, and loss. It drives nft and tc on each node through n.Exec, so it works on any node whose commands have privilege over its network -- a -netns node, a container with CAP_NET_ADMIN, a host reached as root -- and netfault.Check(ctx, nodes) says whether a set of nodes can take faults.

The diskfault package does the same for storage. diskfault.Limit(ctx, n, dir, size) mounts a size-limited tmpfs over a directory -- a service's data directory, before the service first writes it -- and Fill then takes every free byte, so the service's next write fails with ENOSPC, until Free gives the space back; Unlimit removes the limit. A -netns run's nodes may mount it, since they live in a mount namespace the lab's user namespace owns; so may a host reached as root. diskfault.Corrupt(ctx, n, path, offset, size) overwrites part of a file with random bytes, as a disk that silently returns garbage for a sector would, on any node, at rest or under the process using the file.

Local nodes with cgroups of their own (-cgroups). A machine can fail without going down: it freezes for a while, runs out of CPU, swaps, or its disk slows to a crawl. On Linux with a systemd user manager, -cgroups runs every command of each local node in a cgroup (v2) of its own, so a job can do that to one node. No root is needed: the driver re-executes itself with systemd-run --user --scope -p Delegate=yes, a cgroup subtree it owns, puts itself in driver/ there and each node in node-<i>/, and starts every command of a node in the node's cgroup. It combines with -netns: the lab lives inside the scope.

go run ./path/to/suite -cgroups -netns -nodes 6 'my\.chaos'

The resfault package takes those resources away: resfault.Freeze/Thaw stop every process of a node at once (unlike SIGSTOP, unseen by the processes; a command started on a frozen node waits for the thaw, or for its context to end), LimitCPU(n, 0.05) caps a node at a twentieth of a CPU, LimitMemory throttles and reclaims a node's memory past a bound, ThrottleIO(n, dir, limits) bounds the bandwidth and operations per second of the block device under a directory -- a slow disk -- and Reset lifts them all. resfault.Check(n) says whether a node can take them. A slow disk needs a directory on a block device, not a tmpfs, and on a filesystem that compresses, data that compresses well barely reaches the device.

A nemesis, and a checker. A chaos job injects faults while clients load the system, records what each client saw, and checks the record. The nemesis package is the loop that injects the faults: given faults by name and weight, each an Inject that returns how to heal it, (*nemesis.Nemesis).Run(ctx) draws one, holds it, heals it, rests, and goes again until ctx ends, heals the last fault even then, and reports every fault and heal to a Record hook, for the job's history. nemesis.Holds keeps track of what a fault holds down, so that a supervisor -- the nemesis's Before hook -- restarts only what exited on its own.

The linearize package checks the record. Given a model of the object -- an initial state and a step function -- and the operations clients issued, each with its call and return time, linearize.Check says whether some order of the operations, consistent with their real-time order, is a legal sequential run of the model: whether the object was linearizable, as a strongly consistent store promises. An operation whose answer was lost returns at linearize.Pending, and may take effect any time after its call, or never. linearize.Partition splits a history by key, CheckWithin bounds a search's memory, and linearize.Minimize shrinks an illegal history to its core: the few operations that, with every other one weakened, still admit no order -- usually a write, a read that saw it, and a read that did not.

The listappend package checks a record of transactions, after Elle: transactions read lists and append unique elements to them, so the reads of a key reveal the order of its versions, and from that order every dependency between transactions -- write-write, write-read, and read-write (anti-dependencies), and with Options{Realtime: true} real time. listappend.Check returns each kind of anomaly it finds, a cycle as its witness: G0, G1c, G-single, G-nonadjacent, G2; and what needs no cycle: aborted and intermediate reads, reads that disagree on a key's order or with a transaction's own appends, and committed appends a later read missed. listappend.Forbidden(level) says which kinds an isolation level -- serializable, snapshot isolation, read committed -- forbids.

A store the job serves. Some systems keep their data in an object store. The objstore package is one a job runs in its worker: an in-memory server of the part of S3's HTTP API that object-store clients use, with conditional writes, whose faults the job injects and whose every request it records. objstore.New(opts) and (*Server).Serve(addr) start it; give it an address the nodes reach, n.CallerAddr(), which is the worker's address on the route to a node -- the loopback, the -netns lab's bridge, or the interface facing a remote node. AddRule injects faults into the requests of the clients (told apart by the access key they sign with), operations, and key prefixes a rule names, with a probability: a failure or a reset before the request takes effect or after, a delay the client may give up on, a stall that lands after it did, a hang. History returns every request with the fault it got and what it did, and WriteTar archives the objects; both are final once Close returns, which waits out the requests in flight. objstore/cmd/objstore serves one on its own, with an HTTP API for rules, for reproducer scripts.

Because a suite is one static binary, production and multi-node runs invoke it directly; go run/go test is one way to invoke the same binary, not a second code path.

Testing your service and job

Cross-language wire compatibility and full service lifecycles need a live server, so a suite is usually exercised by running it end to end and asserting the result. The idiom (see examples/tutorial/02-service/tutorial_test.go) is a Go test whose TestMain lets the test binary double as the torx worker, then runs the suite through the real driver/worker split on a small local pool:

func TestMain(m *testing.M) {
	if len(os.Args) > 1 && os.Args[1] == "worker" {
		torx.Main() // dispatch to worker mode and exit
	}
	os.Exit(m.Run())
}

func TestSmoke(t *testing.T) {
	reqs, err := torx.Discover("my.smoke")
	// ... build a local pool, torx.Run(ctx, pool, torx.SelfExecLauncher{}, reqs, opts),
	//     assert res.Ok() and inspect the collected results tree ...
}

Everything else is ordinary Go unit-testable, and should be: the pure functions a service and job are built from, such as readiness predicates, address handling, and result parsing. Design services and jobs so their logic is reachable without a running server wherever possible.