diff --git a/.gitignore b/.gitignore index 862de93..32f2dae 100644 --- a/.gitignore +++ b/.gitignore @@ -31,7 +31,6 @@ build*/ # Executables *.exe *.out -!benches/*/leonardo/logs/*.out *.app ### Python ### diff --git a/benches/README.md b/benches/README.md index 051a290..3015dac 100644 --- a/benches/README.md +++ b/benches/README.md @@ -12,8 +12,8 @@ different SRC variants. They are grouped by the kind of workload exercised: wall time on small chains, and quench dynamics of 50 and 100 qubits on the GPU. See [`stack/README.md`](stack/README.md). - [`large/`](large/) — Out-of-core SRC of `N . V . M . U` with a large `M` read - from disk, on one GPU: plan, wall time, memory peaks and stall time. See - [`large/README.md`](large/README.md). + from disk, on one GPU or several: plan, wall time, memory peaks, stall time and + parallel efficiency. See [`large/README.md`](large/README.md). The scripts need the `bench` dependency group (included in `dev`): @@ -21,29 +21,9 @@ The scripts need the `bench` dependency group (included in `dev`): uv sync --group bench ``` -## Leonardo +## HPC clusters -Each folder has a `leonardo/` subfolder with a Slurm script for the Booster -partition of [Leonardo](https://docs.hpc.cineca.it/general/getting_started.html) -at CINECA (one node: 32 Xeon 8358 cores, 4 A100-64GB, 512 GB of memory), and the -logs of the recorded runs. Set up the environment once, from the repository -root on a login node: - -```bash -# Keep the uv cache and Python installs out of $HOME. -export UV_CACHE_DIR=$PWD/.uv-cache UV_PYTHON_INSTALL_DIR=$PWD/.uv-python -uv sync --all-groups --extra gpu-nvidia - -# The Booster driver (535) predates CUDA 13: unpack NVIDIA's forward-compat libcuda. -mkdir -p .cuda-compat && cd .cuda-compat -curl -sSfO https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-compat-13-4-615.71.09-1.el8.x86_64.rpm -rpm2cpio cuda-compat-13-4-*.rpm | cpio -idm && cd .. -``` - -The scripts prepend `.cuda-compat/usr/local/cuda-13.4/compat` to -`LD_LIBRARY_PATH` (override with `CUDA_COMPAT`) and keep CuPy's kernel cache in -`.cupy-cache/`. Submit from the `leonardo/` folder, passing the benchmark's own -arguments, e.g. `sbatch run.sh --n-sites 25 --chi-out 1000 --device gpu`. The -default QoS allows long runs but can queue for hours; for jobs under 30 minutes -add `--qos=boost_qos_dbg --time=00:30:00`, which starts quickly but takes at -most two jobs per user. +The recorded results come from the Booster partition of Leonardo at CINECA. The +[HPC clusters](https://algorithmiq.github.io/src-method/hpc/leonardo) section of +the documentation covers the setup and has an example job script; pass it the +benchmark command, e.g. `python benches/large/bench_large.py run ...`. diff --git a/benches/large/README.md b/benches/large/README.md index cfe7df2..f2a4802 100644 --- a/benches/large/README.md +++ b/benches/large/README.md @@ -46,25 +46,21 @@ size on 8xH100 or 8xH200 (and 4xA100 on Leonardo): 1. Every distance is below `1e-10`. 2. The parallel efficiency is at least 80% on all the GPUs of the node. +`scaling` reports two distances. `distance` comes from inner products and cancels +down to about `sqrt(eps)`, near `1.5e-8` in complex128, so it cannot show `1e-10`. +`QR distance` sweeps a QR over the MPO `a - b` (the direct sum of the two outputs) +and resolves differences down to rounding; the acceptance applies to it. + ## Leonardo -The [`leonardo/`](leonardo/) folder holds the Slurm script and the logs; see the -[Leonardo section](../README.md#leonardo) for the environment. Booster nodes have -no local disk and a 10 GB `/tmp`, so the script points `TMPDIR` at -`$CINECA_SCRATCH`, and the stacks live there too (Lustre, not NVMe): +Recorded on Booster nodes of Leonardo at CINECA; see +[HPC clusters](https://algorithmiq.github.io/src-method/hpc/leonardo) for the setup +and a job script. The nodes have no local disk, so the stacks and the spilled +environments live on Lustre scratch rather than NVMe. -```bash -D=$CINECA_SCRATCH/src-large -sbatch run.sh generate $D/m4000 --bond-m 4000 -sbatch run.sh --debug run $D/m4000 --chi-out 2000 --scratch-dir $D/scratch -sbatch run.sh generate $D/m1000 --bond-m 1000 -sbatch run.sh --debug compare $D/m1000 --chi-out 500 --small 4GB --large 60GB \ - --scratch-dir $D/scratch -``` +### One GPU: D_M = 1000, chi_out = 500 -`generate` writes the `D_M = 1000` stack (12 GB) in 51 s. - -`compare` at `D_M = 1000`, `chi_out = 500`, complex128, on one A100-64GB: +`compare --small 4GB --large 60GB`, complex128, on one A100-64GB: | GPU budget | Wall time (s) | Pool (GB) | Planned device peak (GB) | Host peak (GB) | Tiers (device/host/disk) | |---|---|---|---|---|---| @@ -74,3 +70,49 @@ sbatch run.sh --debug compare $D/m1000 --chi-out 500 --small 4GB --large 60GB \ The two plans keep opposite tiers yet give bit-identical outputs (relative distance `0.000e+00`). Spilling 49 of 50 environments to the host costs 18 % of the wall time. Both pools stay within their caps (budget plus 10 % of the card, 6.9 GB). + +### Several GPUs: D_M = 1000, chi_out = 500 + +One node, 4 A100-SXM-64GB fully connected by NVLink, with peer access on every +pair, so each GPU uploads a slice of every core and gathers the rest from its +peers. `scaling --devices 1 2 4`, complex128, every environment on the device: + +| GPUs | Time (s) | Speed-up | Efficiency | QR distance | +|---|---|---|---|---| +| 1 | 109.9 | 1.00 | 100% | | +| 2 | 43.6 | 2.52 | 126% | 2.8e-13 | +| 4 | 32.7 | 3.36 | 84% | 4.4e-13 | + +- The single-GPU time is that of a second SRC run in its job; the first SRC run of + a job takes about 22 s longer, for a reason not isolated. +- The 2-GPU point is superlinear. The plan and tiers are the same on every count, + and the sketch batch drops from 480 to 224 columns per GPU, which may run the + contractions more efficiently; this was not isolated. +- This problem fits on one GPU, so it says little about the reference size. + +### Several GPUs: D_M = 4000, chi_out = 2000 (reference size) + +50 sites, complex128, 184 GB of input: 49 environments of 8.19 GB, 401 GB in all. +Speed-up and efficiency are against the single-GPU run: + +| GPUs | Time (s) | Speed-up | Efficiency | Left-to-right (s) | Right-to-left (s) | Environments device / host / disk | QR distance | +|---|---|---|---|---|---|---|---| +| 1 | 3925.3 | 1.00 | 100% | 794.8 | 3112.1 | 0 / 45 / 4 | | +| 2 | 2348.8 | 1.67 | 84% | 730.6 | 1606.3 | 1 / 35 / 14 | 7.5e-13 (vs 4 GPUs) | +| 4 | 1099.9 | 3.57 | 89% | 240.4 | 844.1 | 1 / 45 / 3 | | + +- Both efficiencies are above the 80% target. +- Before the out-of-memory split, every run at this size failed. The live memory + stayed within the plan, but CuPy's pool held up to 66.7 GB in split blocks and a + 12.6-14.7 GB allocation did not fit under its 67.6 GB cap. Each GPU of the + multi-GPU runs now splits one sketch batch in two and finishes. +- Four GPUs keep only one environment on the device. The plan picks the largest + batches that fit first, and their working set takes each GPU's budget (59.0 GB + planned peak), although a GPU holds only 2.05 GB of each environment. +- Three or four environments go to disk on one and four GPUs: the host budget, 90% + of the available memory (about 470 GB), falls short of the 487-496 GB that + keeping them all in host memory takes. On two GPUs, 14 go to disk with a planned + host peak of 381.7 GB against 462.6 GB on four; why the plan leaves that host + memory unused was not investigated. +- The stall times the sweep logs include waiting for the GPU's own queue, so they + cannot tell slow transfers from a GPU-bound pass. diff --git a/benches/large/bench_large.py b/benches/large/bench_large.py index 59b8a2e..6c8c50c 100644 --- a/benches/large/bench_large.py +++ b/benches/large/bench_large.py @@ -21,7 +21,7 @@ import json import logging import resource -from pathlib import Path # noqa: TC003 (cyclopts reads the annotations at runtime) +from pathlib import Path from time import perf_counter from typing import Annotated @@ -215,6 +215,31 @@ def _distance(a: list[np.ndarray], b: list[np.ndarray]) -> float: return float(np.sqrt(max((aa + bb - 2 * ab).real, 0.0) / aa.real)) +def _qr_distance(a: list[np.ndarray], b: list[np.ndarray]) -> float: + """Relative distance ``|a - b| / |a|`` from a QR sweep over the MPO ``a - b``. + + `_distance` cancels down to about ``sqrt(eps)``. Here ``a - b`` is a direct sum + of the two trains, and the QR sweep keeps rounding relative to ``|a|``, which + resolves differences down to ``eps``. Gauge-free, unlike comparing cores. + """ + n = len(a) + carry = np.ones((1, 1), dtype=np.result_type(a[0], b[0])) + for j, (x, y) in enumerate(zip(a, b)): + x = x[None] if j == 0 else x[:, None] if j == n - 1 else x + y = y[None] if j == 0 else y[:, None] if j == n - 1 else y + if j == 0: + site = np.concatenate([x, -y], axis=1) + elif j == n - 1: + site = np.concatenate([x, y], axis=0) + else: + (la, ra, *phys), (lb, rb, _, _) = x.shape, y.shape + site = np.zeros((la + lb, ra + rb, *phys), dtype=carry.dtype) + site[:la, :ra], site[la:, ra:] = x, y + site = np.tensordot(carry, site, axes=(1, 0)).transpose(0, 2, 3, 1) + carry = np.linalg.qr(site.reshape(-1, site.shape[-1]), mode="r") + return float(np.linalg.norm(carry) / np.sqrt(_inner(a, a).real)) + + @app.command def scaling( directory: Path, @@ -257,17 +282,20 @@ def scaling( out, seconds = _run(directory, chi_out, resources, seed, device) if reference is None: reference, ref_seconds, ref_devices = out, seconds, count - distance = 0.0 + distance = qr_distance = 0.0 else: distance = _distance(reference, out) + qr_distance = _qr_distance(reference, out) efficiency = ref_seconds * ref_devices / (seconds * count) logger.info( - "GPUs %d: %.1f s, speed-up %.2f, efficiency %.0f %%, distance %.3e", + "GPUs %d: %.1f s, speed-up %.2f, efficiency %.0f %%, distance %.3e, " + "QR distance %.3e", count, seconds, ref_seconds / seconds, 100 * efficiency, distance, + qr_distance, ) if results is not None: record = { @@ -275,6 +303,7 @@ def scaling( "seconds": seconds, "efficiency": efficiency, "distance": distance, + "qr_distance": qr_distance, "chi_out": chi_out, } with results.open("a") as f: diff --git a/benches/large/leonardo/logs/large_59625448.out b/benches/large/leonardo/logs/large_59625448.out deleted file mode 100644 index a976da7..0000000 --- a/benches/large/leonardo/logs/large_59625448.out +++ /dev/null @@ -1,15 +0,0 @@ -NODE: lrdn0113, CPUS: 32, ARGS: generate /leonardo_scratch/large/userexternal/rpanades/src-large/m1000 --bond-m 1000 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -Layer N written: bond 4 -Layer V written: bond 4 -Layer M written: bond 1000 -Layer U written: bond 4 -JobID JobName Elapsed MaxRSS MaxDiskRead MaxDiskWrite ExitCode ------------- ---------- ---------- ---------- ------------ ------------ -------- -59625448 large 00:01:01 0:0 -59625448.ba+ batch 00:01:01 0:0 -59625448.ex+ extern 00:01:01 0:0 -59625448.0 python 00:00:51 5.74G 0.01G 0.02M 0:0 diff --git a/benches/large/leonardo/logs/large_compare_59625755.out b/benches/large/leonardo/logs/large_compare_59625755.out deleted file mode 100644 index 483c5ef..0000000 --- a/benches/large/leonardo/logs/large_compare_59625755.out +++ /dev/null @@ -1,28 +0,0 @@ -NODE: lrdn2346, CPUS: 32, ARGS: --debug compare /leonardo_scratch/large/userexternal/rpanades/src-large/m1000 --chi-out 500 --small 4GB --large 60GB --scratch-dir /leonardo_scratch/large/userexternal/rpanades/src-large/scratch -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -Starting SRC: n_sites=50, depth=4, output=mpo, device=cupy -SRC plan: prefetch=1, device peak=3855012864 B, host peak=30958244864 B, disk=0 B, scratch=None, tiers=['device', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host', 'host'], batches (env, sketch, project)=[(480, 0, 480), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (64, 32, 16), (0, 480, 480)] -Left-to-right sweep: 38.243 s -Right-to-left sweep: 69.762 s -SRC stalls: sites 10.108 s, environments 31.839 s -Device pool: 5460450304 B -SRC complete -Run complete: 129.8 s, pool 5460450304 B, host peak 40588197888 B, planned device peak 3855012864 B, planned host peak 30958244864 B, disk 0 B, tiers {'device': 1, 'host': 49, 'disk': 0}, sketch batches [32, 480] -Starting SRC: n_sites=50, depth=4, output=mpo, device=cupy -SRC plan: prefetch=1, device peak=58841624576 B, host peak=3904164864 B, disk=0 B, scratch=None, tiers=['device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device'], batches (env, sketch, project)=[(480, 0, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (480, 480, 480), (0, 480, 480)] -Left-to-right sweep: 13.859 s -Right-to-left sweep: 95.954 s -SRC stalls: sites 13.835 s, environments 0.000 s -Device pool: 60811395072 B -SRC complete -Run complete: 109.9 s, pool 60811395072 B, host peak 40588197888 B, planned device peak 58841624576 B, planned host peak 3904164864 B, disk 0 B, tiers {'device': 50, 'host': 0, 'disk': 0}, sketch batches [480] -Relative distance between the runs: 0.000e+00 -JobID JobName Elapsed MaxRSS MaxDiskRead MaxDiskWrite ExitCode ------------- ---------- ---------- ---------- ------------ ------------ -------- -59625755 large_com+ 00:04:47 0:0 -59625755.ba+ batch 00:04:47 0:0 -59625755.ex+ extern 00:04:47 0:0 -59625755.0 python 00:04:44 0:0 diff --git a/benches/large/leonardo/logs/large_generate_59625937.out b/benches/large/leonardo/logs/large_generate_59625937.out deleted file mode 100644 index 7c1ba1f..0000000 --- a/benches/large/leonardo/logs/large_generate_59625937.out +++ /dev/null @@ -1,15 +0,0 @@ -NODE: lrdn2918, CPUS: 32, ARGS: generate /leonardo_scratch/large/userexternal/rpanades/src-large/m4000 --bond-m 4000 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -Layer N written: bond 4 -Layer V written: bond 4 -Layer M written: bond 4000 -Layer U written: bond 4 -JobID JobName Elapsed MaxRSS MaxDiskRead MaxDiskWrite ExitCode ------------- ---------- ---------- ---------- ------------ ------------ -------- -59625937 large_gen+ 00:11:35 0:0 -59625937.ba+ batch 00:11:35 0:0 -59625937.ex+ extern 00:11:35 0:0 -59625937.0 python 00:11:31 2.42G 0.01G 0.02M 0:0 diff --git a/benches/large/leonardo/run.sh b/benches/large/leonardo/run.sh deleted file mode 100755 index 55d9037..0000000 --- a/benches/large/leonardo/run.sh +++ /dev/null @@ -1,41 +0,0 @@ -#!/usr/bin/env -S bash -l -# Usage, from this folder: sbatch run.sh [--debug] generate|run|compare [options] -# e.g. sbatch run.sh --debug run "$CINECA_SCRATCH/src-large/m4000" --chi-out 2000 -# Add --qos=boost_qos_dbg (30 min, 2 nodes) to sbatch for short test runs. -#SBATCH --nodes=1 -#SBATCH --ntasks-per-node=1 -#SBATCH --job-name=large -#SBATCH --account=EUHPC_D30_139 -#SBATCH --partition=boost_usr_prod -#SBATCH --time=04:00:00 -#SBATCH --cpus-per-task=32 -#SBATCH --gres=gpu:1 -#SBATCH --mem=0 -#SBATCH --exclusive -#SBATCH --output=logs/%x_%j.out - -set -euo pipefail - -export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK -export OMP_PLACES=cores -export OMP_PROC_BIND=spread -# Booster nodes have no local disk and /tmp is a small tmpfs: spill to Lustre scratch. -export TMPDIR=$CINECA_SCRATCH/src-large/tmp -mkdir -p "$TMPDIR" - -REPO=$(git -C "$SLURM_SUBMIT_DIR" rev-parse --show-toplevel) -source "$REPO/.venv/bin/activate" -export CUPY_CACHE_DIR=$REPO/.cupy-cache -# The Booster driver (535) predates CUDA 13: load NVIDIA's forward-compat libcuda. -CUDA_COMPAT=${CUDA_COMPAT:-$REPO/.cuda-compat/usr/local/cuda-13.4/compat} -if [[ -d $CUDA_COMPAT ]]; then - export LD_LIBRARY_PATH=$CUDA_COMPAT${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH} -fi - -echo "NODE: $SLURMD_NODENAME, CPUS: $SLURM_CPUS_PER_TASK, ARGS: $*" -nvidia-smi --query-gpu=name,driver_version --format=csv,noheader - -srun python "$REPO/benches/large/bench_large.py" "$@" - -sacct --format=JobID,JobName,Elapsed,MaxRSS,MaxDiskRead,MaxDiskWrite,ExitCode \ - --units=G --jobs="$SLURM_JOB_ID" diff --git a/benches/primitives/README.md b/benches/primitives/README.md index 33bbc76..a613011 100644 --- a/benches/primitives/README.md +++ b/benches/primitives/README.md @@ -8,13 +8,17 @@ controlled bond dimension, and compare to a `quimb` reference. ## Leonardo The Leonardo [supercomputer](https://docs.hpc.cineca.it/general/getting_started.html) at CINECA is -a pre-exascale Tier-0 EuroHPC system. The [`leonardo/`](leonardo/) folder holds the Slurm -script and the logs; see the [Leonardo section](../README.md#leonardo) for the environment. +a pre-exascale Tier-0 EuroHPC system; see +[HPC clusters](https://algorithmiq.github.io/src-method/hpc/leonardo) for how to run there. ### MPO-MPO benchmark The benchmark contracts two length-`n_sites` matrix product operators (MPOs) whose initial bond dimension is `chi_out` (unless `chi_id` is specified for the second MPO), then compresses the resulting MPO back to bond dimension `chi_out`. One MPO is fully random; the other is an identity MPO perturbed by summing it with another random MPO with small elements, making the contraction highly compressible. The table below compares `quimb` (CPU, `rsvd` compression) with `src` on the CPU and on one GPU: wall time of the contraction-compression, peak host memory of the job (MaxRSS), speedup over `quimb`, and relative distance to the uncompressed product. +```bash +uv run python benches/primitives/bench_mpo_mpo.py --n-sites 25 --chi-out 1000 --compare no --device gpu +``` + One Booster node (32 cores, A100-64GB), complex128 unless noted, `chi_id=4` at `chi=1000`: | Experiment | Library | Time (s) | MaxRSS (GB) | Speedup | Distance to reference | @@ -60,7 +64,7 @@ Before the refactoring, on Booster (32 CPUs) and DCGP (112 CPUs) nodes: ### MPO-MPO strong scaling (v0.3.2, DCGP) -For this benchmark, we fix the problem size to `n_sites=25`, `chi=1000` and `chi_id=4`, and vary the number of CPUs in a single node. We tweak the OpenMP environment variables in the `run.sh` script to optimize performance, for instance `OMP_PLACES` and `OMP_PROC_BIND`, so the NUMA domain is taken into account. We do this both manually and automatically. +For this benchmark, we fix the problem size to `n_sites=25`, `chi=1000` and `chi_id=4`, and vary the number of CPUs in a single node. We tweak the OpenMP environment variables in the job script to optimize performance, for instance `OMP_PLACES` and `OMP_PROC_BIND`, so the NUMA domain is taken into account. We do this both manually and automatically. | CPUs | Time auto (s) | Time maual (s)| |------|-----------------|---------------| diff --git a/benches/primitives/leonardo/bench_mpo_mpo.py b/benches/primitives/bench_mpo_mpo.py similarity index 100% rename from benches/primitives/leonardo/bench_mpo_mpo.py rename to benches/primitives/bench_mpo_mpo.py diff --git a/benches/primitives/leonardo/__init__.py b/benches/primitives/leonardo/__init__.py deleted file mode 100644 index 3c87791..0000000 --- a/benches/primitives/leonardo/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""Benchmarking scripts for Leonardo cluster.""" diff --git a/benches/primitives/leonardo/logs/mpo_20_100_quimb_cpu_59625646.out b/benches/primitives/leonardo/logs/mpo_20_100_quimb_cpu_59625646.out deleted file mode 100644 index bea1898..0000000 --- a/benches/primitives/leonardo/logs/mpo_20_100_quimb_cpu_59625646.out +++ /dev/null @@ -1,25 +0,0 @@ -NODE: lrdn2278, CPUS: 32, ARGS: --n-sites 20 --chi-out 100 --compare yes --run quimb --device cpu -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -0: 2026-10-08 04:34:38,570 INFO __main__ benchmark_start: n_sites=20, chi_out=100, chi_id=4, phys_dim=2, dtype=complex128, run=quimb, compare=yes, device=cpu -0: 2026-10-08 04:34:38,570 INFO __main__ Generating MPOs... -0: 2026-10-08 04:34:38,780 INFO __main__ Computing reference contraction (no compression)... -0: 2026-10-08 04:34:38,911 INFO __main__ Reference contraction took 0.13113203 s -0: 2026-10-08 04:34:38,911 INFO __main__ Computing Quimb's MPO-MPO contraction (with compression)... -0: 2026-10-08 04:34:56,770 INFO __main__ Quimb's contraction-compression took 17.858333241 s -0: 2026-10-08 04:34:57,396 INFO __main__ - Distance to reference: 0.0 (relative 0.0) -0: 2026-10-08 04:34:57,396 INFO __main__ benchmark_end -JobID JobName NCPUS AveCPU Elapsed ExitCode ------------- ---------- ---------- ---------- ---------- -------- -59625646 mpo_20_10+ 32 00:00:53 0:0 -59625646.ba+ batch 32 00:00:53 0:0 -59625646.ex+ extern 32 00:00:53 0:0 -59625646.0 python 32 00:09:58 00:00:50 0:0 -JobID JobName MaxRSS MaxRSSNode ------------- ---------- ---------- ---------- -59625646 mpo_20_10+ -59625646.ba+ batch -59625646.ex+ extern -59625646.0 python 0.73G lrdn2278 diff --git a/benches/primitives/leonardo/logs/mpo_20_100_src_cpu_59626091.out b/benches/primitives/leonardo/logs/mpo_20_100_src_cpu_59626091.out deleted file mode 100644 index 4db44b7..0000000 --- a/benches/primitives/leonardo/logs/mpo_20_100_src_cpu_59626091.out +++ /dev/null @@ -1,32 +0,0 @@ -NODE: lrdn3007, CPUS: 32, ARGS: --n-sites 20 --chi-out 100 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -0: 2026-10-07 15:51:38,906 INFO __main__ benchmark_start: n_sites=20, chi_out=100, chi_id=4, phys_dim=2, dtype=complex128, run=src, compare=yes, device=cpu -0: 2026-10-07 15:51:38,906 INFO __main__ Generating MPOs... -0: 2026-10-07 15:51:39,734 INFO __main__ Computing reference contraction (no compression)... -0: 2026-10-07 15:51:39,867 INFO __main__ Reference contraction took 0.13266686700000002 s -0: 2026-10-07 15:51:39,867 INFO __main__ Computing SRC's MPO-MPO contraction (with compression)... -0: 2026-10-07 15:51:39,867 DEBUG src_method.stack Starting SRC: n_sites=20, depth=2, output=mpo, device=numpy -0: 2026-10-07 15:51:39,879 DEBUG src_method._sweep SRC plan: prefetch=1, device peak=37439744 B, host peak=14089472 B, disk=0 B, scratch=None, tiers=['device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device'], batches (env, sketch, project)=[(96, 0, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (0, 96, 96)] -0: 2026-10-07 15:51:39,999 DEBUG src_method._sweep Left-to-right sweep: 0.120 s -0: 2026-10-07 15:51:40,113 DEBUG src_method._sweep Right-to-left sweep: 0.114 s -0: 2026-10-07 15:51:40,113 DEBUG src_method._sweep SRC stalls: sites 0.000 s, environments 0.000 s -0: 2026-10-07 15:51:40,114 DEBUG src_method._sweep Device pool: 0 B -0: 2026-10-07 15:51:40,114 DEBUG src_method.stack SRC complete -0: 2026-10-07 15:51:40,114 INFO __main__ SRC's contraction-compression took 0.247298909 s -0: 2026-10-07 15:51:40,612 INFO __main__ - Distance to reference: 1.4901161193847656e-08 (relative 1.4901161193847666e-08) -0: 2026-10-07 15:51:40,612 INFO __main__ benchmark_end -JobID JobName NCPUS AveCPU Elapsed ExitCode ------------- ---------- ---------- ---------- ---------- -------- -59626091 mpo_20_10+ 32 00:00:24 0:0 -59626091.ba+ batch 32 00:00:24 0:0 -59626091.ex+ extern 32 00:00:24 0:0 -59626091.0 python 32 00:00:20 0:0 -JobID JobName MaxRSS MaxRSSNode ------------- ---------- ---------- ---------- -59626091 mpo_20_10+ -59626091.ba+ batch -59626091.ex+ extern -59626091.0 python diff --git a/benches/primitives/leonardo/logs/mpo_20_100_src_gpu_59626976.out b/benches/primitives/leonardo/logs/mpo_20_100_src_gpu_59626976.out deleted file mode 100644 index fdfcc88..0000000 --- a/benches/primitives/leonardo/logs/mpo_20_100_src_gpu_59626976.out +++ /dev/null @@ -1,41 +0,0 @@ -NODE: lrdn0473, CPUS: 32, ARGS: --n-sites 20 --chi-out 100 --device gpu -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -0: 2026-10-07 16:03:39,120 INFO __main__ benchmark_start: n_sites=20, chi_out=100, chi_id=4, phys_dim=2, dtype=complex128, run=src, compare=yes, device=gpu -0: 2026-10-07 16:03:39,120 INFO __main__ Generating MPOs... -0: 2026-10-07 16:03:39,298 INFO __main__ Computing reference contraction (no compression)... -0: 2026-10-07 16:03:39,441 INFO __main__ Reference contraction took 0.143532198 s -0: 2026-10-07 16:03:39,441 INFO __main__ Warming up the GPU... -0: 2026-10-07 16:03:56,097 DEBUG src_method.stack Starting SRC: n_sites=4, depth=2, output=mpo, device=cupy -0: 2026-10-07 16:03:57,085 DEBUG src_method._sweep SRC plan: prefetch=1, device peak=4224 B, host peak=2432 B, disk=0 B, scratch=None, tiers=['device', 'device', 'device', 'device'], batches (env, sketch, project)=[(2, 0, 2), (2, 2, 2), (2, 2, 2), (0, 2, 2)] -0: 2026-10-07 16:04:11,566 DEBUG src_method._sweep Left-to-right sweep: 14.230 s -0: 2026-10-07 16:04:23,004 DEBUG src_method._sweep Right-to-left sweep: 11.439 s -0: 2026-10-07 16:04:23,004 DEBUG src_method._sweep SRC stalls: sites 0.000 s, environments 0.000 s -0: 2026-10-07 16:04:23,004 DEBUG src_method._sweep Device pool: 48640 B -0: 2026-10-07 16:04:23,005 DEBUG src_method.stack SRC complete -0: 2026-10-07 16:04:23,005 INFO __main__ Computing SRC's MPO-MPO contraction (with compression)... -0: 2026-10-07 16:04:23,005 DEBUG src_method.stack Starting SRC: n_sites=20, depth=2, output=mpo, device=cupy -0: 2026-10-07 16:04:23,017 DEBUG src_method._sweep SRC plan: prefetch=1, device peak=23350272 B, host peak=14089472 B, disk=0 B, scratch=None, tiers=['device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device'], batches (env, sketch, project)=[(96, 0, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (96, 96, 96), (0, 96, 96)] -0: 2026-10-07 16:04:23,148 DEBUG src_method._sweep Left-to-right sweep: 0.131 s -0: 2026-10-07 16:04:23,459 DEBUG src_method._sweep Right-to-left sweep: 0.311 s -0: 2026-10-07 16:04:23,460 DEBUG src_method._sweep SRC stalls: sites 0.000 s, environments 0.000 s -0: 2026-10-07 16:04:23,460 DEBUG src_method._sweep Device pool: 24724992 B -0: 2026-10-07 16:04:23,460 DEBUG src_method.stack SRC complete -0: 2026-10-07 16:04:23,460 INFO __main__ SRC's contraction-compression took 0.45528803300000004 s -0: 2026-10-07 16:04:23,460 INFO __main__ - CuPy pool high-water mark: 0.02 GB -0: 2026-10-07 16:04:23,978 INFO __main__ - Distance to reference: 2.1073424255447017e-08 (relative 2.107342425544703e-08) -0: 2026-10-07 16:04:23,979 INFO __main__ benchmark_end -JobID JobName NCPUS AveCPU Elapsed ExitCode ------------- ---------- ---------- ---------- ---------- -------- -59626976 mpo_20_10+ 32 00:01:14 0:0 -59626976.ba+ batch 32 00:01:14 0:0 -59626976.ex+ extern 32 00:01:14 0:0 -59626976.0 python 32 00:00:33 00:01:10 0:0 -JobID JobName MaxRSS MaxRSSNode ------------- ---------- ---------- ---------- -59626976 mpo_20_10+ -59626976.ba+ batch -59626976.ex+ extern -59626976.0 python 1.22G lrdn0473 diff --git a/benches/primitives/leonardo/logs/mpo_25_1000_quimb_cpu_59626117.out b/benches/primitives/leonardo/logs/mpo_25_1000_quimb_cpu_59626117.out deleted file mode 100644 index be789e3..0000000 --- a/benches/primitives/leonardo/logs/mpo_25_1000_quimb_cpu_59626117.out +++ /dev/null @@ -1,22 +0,0 @@ -NODE: lrdn2680, CPUS: 32, ARGS: --n-sites 25 --chi-out 1000 --compare no --run quimb -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -0: 2026-10-07 15:52:23,165 INFO __main__ benchmark_start: n_sites=25, chi_out=1000, chi_id=4, phys_dim=2, dtype=complex128, run=quimb, compare=no, device=cpu -0: 2026-10-07 15:52:23,165 INFO __main__ Generating MPOs... -0: 2026-10-07 15:52:28,198 INFO __main__ Computing Quimb's MPO-MPO contraction (with compression)... -0: 2026-10-07 15:59:30,772 INFO __main__ Quimb's contraction-compression took 422.574737481 s -0: 2026-10-07 15:59:30,773 INFO __main__ benchmark_end -JobID JobName NCPUS AveCPU Elapsed ExitCode ------------- ---------- ---------- ---------- ---------- -------- -59626117 mpo_25_10+ 32 00:07:31 0:0 -59626117.ba+ batch 32 00:07:31 0:0 -59626117.ex+ extern 32 00:07:31 0:0 -59626117.0 python 32 03:17:05 00:07:27 0:0 -JobID JobName MaxRSS MaxRSSNode ------------- ---------- ---------- ---------- -59626117 mpo_25_10+ -59626117.ba+ batch -59626117.ex+ extern -59626117.0 python 24.64G lrdn2680 diff --git a/benches/primitives/leonardo/logs/mpo_25_1000_src_cpu_59626629.out b/benches/primitives/leonardo/logs/mpo_25_1000_src_cpu_59626629.out deleted file mode 100644 index 2ab181b..0000000 --- a/benches/primitives/leonardo/logs/mpo_25_1000_src_cpu_59626629.out +++ /dev/null @@ -1,29 +0,0 @@ -NODE: lrdn2680, CPUS: 32, ARGS: --n-sites 25 --chi-out 1000 --compare no -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -0: 2026-10-07 16:00:06,993 INFO __main__ benchmark_start: n_sites=25, chi_out=1000, chi_id=4, phys_dim=2, dtype=complex128, run=src, compare=no, device=cpu -0: 2026-10-07 16:00:06,993 INFO __main__ Generating MPOs... -0: 2026-10-07 16:00:12,023 INFO __main__ Computing SRC's MPO-MPO contraction (with compression)... -0: 2026-10-07 16:00:12,023 DEBUG src_method.stack Starting SRC: n_sites=25, depth=2, output=mpo, device=numpy -0: 2026-10-07 16:00:12,045 DEBUG src_method._sweep SRC plan: prefetch=1, device peak=4409414144 B, host peak=1728067072 B, disk=0 B, scratch=None, tiers=['device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device'], batches (env, sketch, project)=[(992, 0, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (0, 992, 992)] -0: 2026-10-07 16:00:31,730 DEBUG src_method._sweep Left-to-right sweep: 19.685 s -0: 2026-10-07 16:01:12,496 DEBUG src_method._sweep Right-to-left sweep: 40.766 s -0: 2026-10-07 16:01:12,496 DEBUG src_method._sweep SRC stalls: sites 0.000 s, environments 0.000 s -0: 2026-10-07 16:01:12,496 DEBUG src_method._sweep Device pool: 0 B -0: 2026-10-07 16:01:12,497 DEBUG src_method.stack SRC complete -0: 2026-10-07 16:01:12,497 INFO __main__ SRC's contraction-compression took 60.47406152000001 s -0: 2026-10-07 16:01:12,497 INFO __main__ benchmark_end -JobID JobName NCPUS AveCPU Elapsed ExitCode ------------- ---------- ---------- ---------- ---------- -------- -59626629 mpo_25_10+ 32 00:01:30 0:0 -59626629.ba+ batch 32 00:01:30 0:0 -59626629.ex+ extern 32 00:01:30 0:0 -59626629.0 python 32 00:01:26 0:0 -JobID JobName MaxRSS MaxRSSNode ------------- ---------- ---------- ---------- -59626629 mpo_25_10+ -59626629.ba+ batch -59626629.ex+ extern -59626629.0 python 3.85G lrdn2680 diff --git a/benches/primitives/leonardo/logs/mpo_25_1000_src_gpu_59627165.out b/benches/primitives/leonardo/logs/mpo_25_1000_src_gpu_59627165.out deleted file mode 100644 index 8fa0f82..0000000 --- a/benches/primitives/leonardo/logs/mpo_25_1000_src_gpu_59627165.out +++ /dev/null @@ -1,38 +0,0 @@ -NODE: lrdn2796, CPUS: 32, ARGS: --n-sites 25 --chi-out 1000 --compare no --device gpu -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -0: 2026-10-07 16:05:28,788 INFO __main__ benchmark_start: n_sites=25, chi_out=1000, chi_id=4, phys_dim=2, dtype=complex128, run=src, compare=no, device=gpu -0: 2026-10-07 16:05:28,788 INFO __main__ Generating MPOs... -0: 2026-10-07 16:05:34,012 INFO __main__ Warming up the GPU... -0: 2026-10-07 16:05:54,418 DEBUG src_method.stack Starting SRC: n_sites=4, depth=2, output=mpo, device=cupy -0: 2026-10-07 16:05:55,873 DEBUG src_method._sweep SRC plan: prefetch=1, device peak=4224 B, host peak=2432 B, disk=0 B, scratch=None, tiers=['device', 'device', 'device', 'device'], batches (env, sketch, project)=[(2, 0, 2), (2, 2, 2), (2, 2, 2), (0, 2, 2)] -0: 2026-10-07 16:06:12,531 DEBUG src_method._sweep Left-to-right sweep: 16.401 s -0: 2026-10-07 16:06:21,998 DEBUG src_method._sweep Right-to-left sweep: 9.467 s -0: 2026-10-07 16:06:21,998 DEBUG src_method._sweep SRC stalls: sites 0.000 s, environments 0.000 s -0: 2026-10-07 16:06:21,998 DEBUG src_method._sweep Device pool: 48640 B -0: 2026-10-07 16:06:21,998 DEBUG src_method.stack SRC complete -0: 2026-10-07 16:06:21,998 INFO __main__ Computing SRC's MPO-MPO contraction (with compression)... -0: 2026-10-07 16:06:21,998 DEBUG src_method.stack Starting SRC: n_sites=25, depth=2, output=mpo, device=cupy -0: 2026-10-07 16:06:22,015 DEBUG src_method._sweep SRC plan: prefetch=1, device peak=2681347072 B, host peak=1728067072 B, disk=0 B, scratch=None, tiers=['device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device'], batches (env, sketch, project)=[(992, 0, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (0, 992, 992)] -0: 2026-10-07 16:06:22,462 DEBUG src_method._sweep Left-to-right sweep: 0.406 s -0: 2026-10-07 16:06:24,322 DEBUG src_method._sweep Right-to-left sweep: 1.859 s -0: 2026-10-07 16:06:24,322 DEBUG src_method._sweep SRC stalls: sites 0.188 s, environments 0.000 s -0: 2026-10-07 16:06:24,322 DEBUG src_method._sweep Device pool: 2795244544 B -0: 2026-10-07 16:06:24,322 DEBUG src_method.stack SRC complete -0: 2026-10-07 16:06:24,323 INFO __main__ SRC's contraction-compression took 2.3248113050000003 s -0: 2026-10-07 16:06:24,323 INFO __main__ - CuPy pool high-water mark: 2.80 GB -0: 2026-10-07 16:06:24,323 INFO __main__ benchmark_end -JobID JobName NCPUS AveCPU Elapsed ExitCode ------------- ---------- ---------- ---------- ---------- -------- -59627165 mpo_25_10+ 32 00:01:24 0:0 -59627165.ba+ batch 32 00:01:24 0:0 -59627165.ex+ extern 32 00:01:24 0:0 -59627165.0 python 32 00:01:21 0:0 -JobID JobName MaxRSS MaxRSSNode ------------- ---------- ---------- ---------- -59627165 mpo_25_10+ -59627165.ba+ batch -59627165.ex+ extern -59627165.0 python diff --git a/benches/primitives/leonardo/logs/mpo_25_1000_src_gpu_c64_59627591.out b/benches/primitives/leonardo/logs/mpo_25_1000_src_gpu_c64_59627591.out deleted file mode 100644 index 9e3d140..0000000 --- a/benches/primitives/leonardo/logs/mpo_25_1000_src_gpu_c64_59627591.out +++ /dev/null @@ -1,38 +0,0 @@ -NODE: lrdn0096, CPUS: 32, ARGS: --n-sites 25 --chi-out 1000 --compare no --device gpu --dtype complex64 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -0: 2026-10-07 16:09:10,881 INFO __main__ benchmark_start: n_sites=25, chi_out=1000, chi_id=4, phys_dim=2, dtype=complex64, run=src, compare=no, device=gpu -0: 2026-10-07 16:09:10,881 INFO __main__ Generating MPOs... -0: 2026-10-07 16:09:13,471 INFO __main__ Warming up the GPU... -0: 2026-10-07 16:09:34,542 DEBUG src_method.stack Starting SRC: n_sites=4, depth=2, output=mpo, device=cupy -0: 2026-10-07 16:09:35,964 DEBUG src_method._sweep SRC plan: prefetch=1, device peak=2112 B, host peak=1216 B, disk=0 B, scratch=None, tiers=['device', 'device', 'device', 'device'], batches (env, sketch, project)=[(2, 0, 2), (2, 2, 2), (2, 2, 2), (0, 2, 2)] -0: 2026-10-07 16:09:57,185 DEBUG src_method._sweep Left-to-right sweep: 20.986 s -0: 2026-10-07 16:10:09,659 DEBUG src_method._sweep Right-to-left sweep: 12.474 s -0: 2026-10-07 16:10:09,659 DEBUG src_method._sweep SRC stalls: sites 0.000 s, environments 0.000 s -0: 2026-10-07 16:10:09,659 DEBUG src_method._sweep Device pool: 27648 B -0: 2026-10-07 16:10:09,659 DEBUG src_method.stack SRC complete -0: 2026-10-07 16:10:09,659 INFO __main__ Computing SRC's MPO-MPO contraction (with compression)... -0: 2026-10-07 16:10:09,659 DEBUG src_method.stack Starting SRC: n_sites=25, depth=2, output=mpo, device=cupy -0: 2026-10-07 16:10:09,676 DEBUG src_method._sweep SRC plan: prefetch=1, device peak=1340673536 B, host peak=864033536 B, disk=0 B, scratch=None, tiers=['device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device'], batches (env, sketch, project)=[(992, 0, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (0, 992, 992)] -0: 2026-10-07 16:10:10,116 DEBUG src_method._sweep Left-to-right sweep: 0.418 s -0: 2026-10-07 16:10:11,554 DEBUG src_method._sweep Right-to-left sweep: 1.438 s -0: 2026-10-07 16:10:11,554 DEBUG src_method._sweep SRC stalls: sites 0.146 s, environments 0.000 s -0: 2026-10-07 16:10:11,554 DEBUG src_method._sweep Device pool: 1397658112 B -0: 2026-10-07 16:10:11,554 DEBUG src_method.stack SRC complete -0: 2026-10-07 16:10:11,555 INFO __main__ SRC's contraction-compression took 1.895542328 s -0: 2026-10-07 16:10:11,555 INFO __main__ - CuPy pool high-water mark: 1.40 GB -0: 2026-10-07 16:10:11,555 INFO __main__ benchmark_end -JobID JobName NCPUS AveCPU Elapsed ExitCode ------------- ---------- ---------- ---------- ---------- -------- -59627591 mpo_25_10+ 32 00:01:32 0:0 -59627591.ba+ batch 32 00:01:32 0:0 -59627591.ex+ extern 32 00:01:32 0:0 -59627591.0 python 32 00:00:50 00:01:26 0:0 -JobID JobName MaxRSS MaxRSSNode ------------- ---------- ---------- ---------- -59627591 mpo_25_10+ -59627591.ba+ batch -59627591.ex+ extern -59627591.0 python 2.20G lrdn0096 diff --git a/benches/primitives/leonardo/logs/mpo_50_1000_quimb_cpu_59625936.out b/benches/primitives/leonardo/logs/mpo_50_1000_quimb_cpu_59625936.out deleted file mode 100644 index 5d6d519..0000000 --- a/benches/primitives/leonardo/logs/mpo_50_1000_quimb_cpu_59625936.out +++ /dev/null @@ -1,22 +0,0 @@ -NODE: lrdn2427, CPUS: 32, ARGS: --n-sites 50 --chi-out 1000 --compare no --run quimb -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -0: 2026-10-08 04:35:27,338 INFO __main__ benchmark_start: n_sites=50, chi_out=1000, chi_id=4, phys_dim=2, dtype=complex128, run=quimb, compare=no, device=cpu -0: 2026-10-08 04:35:27,338 INFO __main__ Generating MPOs... -0: 2026-10-08 04:35:38,108 INFO __main__ Computing Quimb's MPO-MPO contraction (with compression)... -0: 2026-10-08 04:52:58,831 INFO __main__ Quimb's contraction-compression took 1040.7233631410002 s -0: 2026-10-08 04:52:58,831 INFO __main__ benchmark_end -JobID JobName NCPUS AveCPU Elapsed ExitCode ------------- ---------- ---------- ---------- ---------- -------- -59625936 mpo_50_10+ 32 00:18:05 0:0 -59625936.ba+ batch 32 00:18:05 0:0 -59625936.ex+ extern 32 00:18:05 0:0 -59625936.0 python 32 00:18:02 0:0 -JobID JobName MaxRSS MaxRSSNode ------------- ---------- ---------- ---------- -59625936 mpo_50_10+ -59625936.ba+ batch -59625936.ex+ extern -59625936.0 python diff --git a/benches/primitives/leonardo/logs/mpo_50_1000_src_cpu_59626808.out b/benches/primitives/leonardo/logs/mpo_50_1000_src_cpu_59626808.out deleted file mode 100644 index 593270b..0000000 --- a/benches/primitives/leonardo/logs/mpo_50_1000_src_cpu_59626808.out +++ /dev/null @@ -1,30 +0,0 @@ -NODE: lrdn2517, CPUS: 32, ARGS: --n-sites 50 --chi-out 1000 --compare no -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -0: 2026-10-07 16:02:04,119 INFO __main__ benchmark_start: n_sites=50, chi_out=1000, chi_id=4, phys_dim=2, dtype=complex128, run=src, compare=no, device=cpu -0: 2026-10-07 16:02:04,119 INFO __main__ Generating MPOs... -0: 2026-10-07 16:02:15,292 INFO __main__ Computing SRC's MPO-MPO contraction (with compression)... -0: 2026-10-07 16:02:15,293 DEBUG src_method.stack Starting SRC: n_sites=50, depth=2, output=mpo, device=numpy -0: 2026-10-07 16:02:15,328 DEBUG src_method._sweep SRC plan: prefetch=1, device peak=7609414144 B, host peak=3328067072 B, disk=0 B, scratch=None, tiers=['device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device'], batches (env, sketch, project)=[(992, 0, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (9 -0: 92, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (0, 992, 992)] -0: 2026-10-07 16:02:56,795 DEBUG src_method._sweep Left-to-right sweep: 41.467 s -0: 2026-10-07 16:04:32,664 DEBUG src_method._sweep Right-to-left sweep: 95.868 s -0: 2026-10-07 16:04:32,664 DEBUG src_method._sweep SRC stalls: sites 0.001 s, environments 0.000 s -0: 2026-10-07 16:04:32,664 DEBUG src_method._sweep Device pool: 0 B -0: 2026-10-07 16:04:32,665 DEBUG src_method.stack SRC complete -0: 2026-10-07 16:04:32,665 INFO __main__ SRC's contraction-compression took 137.37281321700002 s -0: 2026-10-07 16:04:32,665 INFO __main__ benchmark_end -JobID JobName NCPUS AveCPU Elapsed ExitCode ------------- ---------- ---------- ---------- ---------- -------- -59626808 mpo_50_10+ 32 00:02:52 0:0 -59626808.ba+ batch 32 00:02:52 0:0 -59626808.ex+ extern 32 00:02:52 0:0 -59626808.0 python 32 00:50:14 00:02:48 0:0 -JobID JobName MaxRSS MaxRSSNode ------------- ---------- ---------- ---------- -59626808 mpo_50_10+ -59626808.ba+ batch -59626808.ex+ extern -59626808.0 python 6.83G lrdn2517 diff --git a/benches/primitives/leonardo/logs/mpo_50_1000_src_gpu_59627373.out b/benches/primitives/leonardo/logs/mpo_50_1000_src_gpu_59627373.out deleted file mode 100644 index 312b3f2..0000000 --- a/benches/primitives/leonardo/logs/mpo_50_1000_src_gpu_59627373.out +++ /dev/null @@ -1,39 +0,0 @@ -NODE: lrdn3032, CPUS: 32, ARGS: --n-sites 50 --chi-out 1000 --compare no --device gpu -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -0: 2026-10-07 16:07:18,364 INFO __main__ benchmark_start: n_sites=50, chi_out=1000, chi_id=4, phys_dim=2, dtype=complex128, run=src, compare=no, device=gpu -0: 2026-10-07 16:07:18,364 INFO __main__ Generating MPOs... -0: 2026-10-07 16:07:29,145 INFO __main__ Warming up the GPU... -0: 2026-10-07 16:07:42,169 DEBUG src_method.stack Starting SRC: n_sites=4, depth=2, output=mpo, device=cupy -0: 2026-10-07 16:07:43,471 DEBUG src_method._sweep SRC plan: prefetch=1, device peak=4224 B, host peak=2432 B, disk=0 B, scratch=None, tiers=['device', 'device', 'device', 'device'], batches (env, sketch, project)=[(2, 0, 2), (2, 2, 2), (2, 2, 2), (0, 2, 2)] -0: 2026-10-07 16:07:59,920 DEBUG src_method._sweep Left-to-right sweep: 16.131 s -0: 2026-10-07 16:08:11,732 DEBUG src_method._sweep Right-to-left sweep: 11.812 s -0: 2026-10-07 16:08:11,732 DEBUG src_method._sweep SRC stalls: sites 0.000 s, environments 0.000 s -0: 2026-10-07 16:08:11,732 DEBUG src_method._sweep Device pool: 48640 B -0: 2026-10-07 16:08:11,733 DEBUG src_method.stack SRC complete -0: 2026-10-07 16:08:11,733 INFO __main__ Computing SRC's MPO-MPO contraction (with compression)... -0: 2026-10-07 16:08:11,733 DEBUG src_method.stack Starting SRC: n_sites=50, depth=2, output=mpo, device=cupy -0: 2026-10-07 16:08:11,752 DEBUG src_method._sweep SRC plan: prefetch=1, device peak=4281347072 B, host peak=3328067072 B, disk=0 B, scratch=None, tiers=['device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device'], batches (env, sketch, project)=[(992, 0, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (9 -0: 92, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (0, 992, 992)] -0: 2026-10-07 16:08:12,392 DEBUG src_method._sweep Left-to-right sweep: 0.604 s -0: 2026-10-07 16:08:16,200 DEBUG src_method._sweep Right-to-left sweep: 3.809 s -0: 2026-10-07 16:08:16,200 DEBUG src_method._sweep SRC stalls: sites 0.398 s, environments 0.000 s -0: 2026-10-07 16:08:16,200 DEBUG src_method._sweep Device pool: 5080812544 B -0: 2026-10-07 16:08:16,201 DEBUG src_method.stack SRC complete -0: 2026-10-07 16:08:16,201 INFO __main__ SRC's contraction-compression took 4.468574902 s -0: 2026-10-07 16:08:16,201 INFO __main__ - CuPy pool high-water mark: 5.08 GB -0: 2026-10-07 16:08:16,201 INFO __main__ benchmark_end -JobID JobName NCPUS AveCPU Elapsed ExitCode ------------- ---------- ---------- ---------- ---------- -------- -59627373 mpo_50_10+ 32 00:01:30 0:0 -59627373.ba+ batch 32 00:01:30 0:0 -59627373.ex+ extern 32 00:01:30 0:0 -59627373.0 python 32 00:05:16 00:01:26 0:0 -JobID JobName MaxRSS MaxRSSNode ------------- ---------- ---------- ---------- -59627373 mpo_50_10+ -59627373.ba+ batch -59627373.ex+ extern -59627373.0 python 6.40G lrdn3032 diff --git a/benches/primitives/leonardo/logs/mpo_50_1000_src_gpu_c64_59627813.out b/benches/primitives/leonardo/logs/mpo_50_1000_src_gpu_c64_59627813.out deleted file mode 100644 index 58d0fc1..0000000 --- a/benches/primitives/leonardo/logs/mpo_50_1000_src_gpu_c64_59627813.out +++ /dev/null @@ -1,45 +0,0 @@ -NODE: lrdn0096, CPUS: 32, ARGS: --n-sites 50 --chi-out 1000 --compare no --device gpu --dtype complex64 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -0: 2026-10-07 16:10:38,169 INFO __main__ benchmark_start: n_sites=50, chi_out=1000, chi_id=4, phys_dim=2, dtype=complex64, run=src, compare=no, device=gpu -0: 2026-10-07 16:10:38,169 INFO __main__ Generating MPOs... -0: /leonardo_work/ALQ_prod2526/rpanades/src-method/.venv/lib/python3.14/site-packages/autoray/autoray.py:96: RuntimeWarning: overflow encountered in matmul -0: return func(*args, **kwargs) -0: /leonardo_work/ALQ_prod2526/rpanades/src-method/.venv/lib/python3.14/site-packages/autoray/autoray.py:96: RuntimeWarning: invalid value encountered in matmul -0: return func(*args, **kwargs) -0: /leonardo_work/ALQ_prod2526/rpanades/src-method/.venv/lib/python3.14/site-packages/quimb/tensor/tensor_core.py:5119: RuntimeWarning: invalid value encountered in scalar power -0: return self.multiply_(other**-1) -0: 2026-10-07 16:10:42,593 INFO __main__ Warming up the GPU... -0: 2026-10-07 16:10:42,861 DEBUG src_method.stack Starting SRC: n_sites=4, depth=2, output=mpo, device=cupy -0: 2026-10-07 16:10:42,960 DEBUG src_method._sweep SRC plan: prefetch=1, device peak=2112 B, host peak=1216 B, disk=0 B, scratch=None, tiers=['device', 'device', 'device', 'device'], batches (env, sketch, project)=[(2, 0, 2), (2, 2, 2), (2, 2, 2), (0, 2, 2)] -0: 2026-10-07 16:10:43,041 DEBUG src_method._sweep Left-to-right sweep: 0.049 s -0: 2026-10-07 16:10:43,069 DEBUG src_method._sweep Right-to-left sweep: 0.027 s -0: 2026-10-07 16:10:43,069 DEBUG src_method._sweep SRC stalls: sites 0.000 s, environments 0.000 s -0: 2026-10-07 16:10:43,069 DEBUG src_method._sweep Device pool: 27648 B -0: 2026-10-07 16:10:43,069 DEBUG src_method.stack SRC complete -0: 2026-10-07 16:10:43,069 INFO __main__ Computing SRC's MPO-MPO contraction (with compression)... -0: 2026-10-07 16:10:43,069 DEBUG src_method.stack Starting SRC: n_sites=50, depth=2, output=mpo, device=cupy -0: 2026-10-07 16:10:43,088 DEBUG src_method._sweep SRC plan: prefetch=1, device peak=2140673536 B, host peak=1664033536 B, disk=0 B, scratch=None, tiers=['device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device'], batches (env, sketch, project)=[(992, 0, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (9 -0: 92, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (992, 992, 992), (0, 992, 992)] -0: 2026-10-07 16:10:43,492 DEBUG src_method._sweep Left-to-right sweep: 0.385 s -0: 2026-10-07 16:10:46,100 DEBUG src_method._sweep Right-to-left sweep: 2.608 s -0: 2026-10-07 16:10:46,100 DEBUG src_method._sweep SRC stalls: sites 0.326 s, environments 0.000 s -0: 2026-10-07 16:10:46,100 DEBUG src_method._sweep Device pool: 2540442112 B -0: 2026-10-07 16:10:46,101 DEBUG src_method.stack SRC complete -0: 2026-10-07 16:10:46,101 INFO __main__ SRC's contraction-compression took 3.032048248 s -0: 2026-10-07 16:10:46,101 INFO __main__ - CuPy pool high-water mark: 2.54 GB -0: 2026-10-07 16:10:46,101 INFO __main__ benchmark_end -JobID JobName NCPUS AveCPU Elapsed ExitCode ------------- ---------- ---------- ---------- ---------- -------- -59627813 mpo_50_10+ 32 00:00:13 0:0 -59627813.ba+ batch 32 00:00:13 0:0 -59627813.ex+ extern 32 00:00:13 0:0 -59627813.0 python 32 00:00:10 0:0 -JobID JobName MaxRSS MaxRSSNode ------------- ---------- ---------- ---------- -59627813 mpo_50_10+ -59627813.ba+ batch -59627813.ex+ extern -59627813.0 python diff --git a/benches/primitives/leonardo/logs/mpo_50_50_quimb_cpu_59625938.out b/benches/primitives/leonardo/logs/mpo_50_50_quimb_cpu_59625938.out deleted file mode 100644 index 135acb3..0000000 --- a/benches/primitives/leonardo/logs/mpo_50_50_quimb_cpu_59625938.out +++ /dev/null @@ -1,25 +0,0 @@ -NODE: lrdn2926, CPUS: 32, ARGS: --n-sites 50 --chi-out 50 --run quimb -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -0: 2026-10-07 15:49:42,313 INFO __main__ benchmark_start: n_sites=50, chi_out=50, chi_id=4, phys_dim=2, dtype=complex128, run=quimb, compare=yes, device=cpu -0: 2026-10-07 15:49:42,314 INFO __main__ Generating MPOs... -0: 2026-10-07 15:49:42,502 INFO __main__ Computing reference contraction (no compression)... -0: 2026-10-07 15:49:42,557 INFO __main__ Reference contraction took 0.05452135 s -0: 2026-10-07 15:49:42,557 INFO __main__ Computing Quimb's MPO-MPO contraction (with compression)... -0: 2026-10-07 15:50:20,313 INFO __main__ Quimb's contraction-compression took 37.755962594 s -0: 2026-10-07 15:50:20,696 INFO __main__ - Distance to reference: 2.5809568279517847e-08 (relative 2.5809568279517943e-08) -0: 2026-10-07 15:50:20,696 INFO __main__ benchmark_end -JobID JobName NCPUS AveCPU Elapsed ExitCode ------------- ---------- ---------- ---------- ---------- -------- -59625938 mpo_50_50+ 32 00:01:04 0:0 -59625938.ba+ batch 32 00:01:04 0:0 -59625938.ex+ extern 32 00:01:04 0:0 -59625938.0 python 32 00:16:51 00:01:00 0:0 -JobID JobName MaxRSS MaxRSSNode ------------- ---------- ---------- ---------- -59625938 mpo_50_50+ -59625938.ba+ batch -59625938.ex+ extern -59625938.0 python 0.67G lrdn2926 diff --git a/benches/primitives/leonardo/logs/mpo_50_50_src_cpu_59626058.out b/benches/primitives/leonardo/logs/mpo_50_50_src_cpu_59626058.out deleted file mode 100644 index a2b35e9..0000000 --- a/benches/primitives/leonardo/logs/mpo_50_50_src_cpu_59626058.out +++ /dev/null @@ -1,33 +0,0 @@ -NODE: lrdn1912, CPUS: 32, ARGS: --n-sites 50 --chi-out 50 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -0: 2026-10-07 15:50:58,923 INFO __main__ benchmark_start: n_sites=50, chi_out=50, chi_id=4, phys_dim=2, dtype=complex128, run=src, compare=yes, device=cpu -0: 2026-10-07 15:50:58,923 INFO __main__ Generating MPOs... -0: 2026-10-07 15:50:59,079 INFO __main__ Computing reference contraction (no compression)... -0: 2026-10-07 15:50:59,134 INFO __main__ Reference contraction took 0.054777736 s -0: 2026-10-07 15:50:59,134 INFO __main__ Computing SRC's MPO-MPO contraction (with compression)... -0: 2026-10-07 15:50:59,134 DEBUG src_method.stack Starting SRC: n_sites=50, depth=2, output=mpo, device=numpy -0: 2026-10-07 15:50:59,149 DEBUG src_method._sweep SRC plan: prefetch=1, device peak=18300544 B, host peak=8326272 B, disk=0 B, scratch=None, tiers=['device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device'], batches (env, sketch, project)=[(32, 0, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 3 -0: 2), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (0, 32, 32)] -0: 2026-10-07 15:50:59,239 DEBUG src_method._sweep Left-to-right sweep: 0.090 s -0: 2026-10-07 15:50:59,364 DEBUG src_method._sweep Right-to-left sweep: 0.125 s -0: 2026-10-07 15:50:59,364 DEBUG src_method._sweep SRC stalls: sites 0.000 s, environments 0.000 s -0: 2026-10-07 15:50:59,364 DEBUG src_method._sweep Device pool: 0 B -0: 2026-10-07 15:50:59,365 DEBUG src_method.stack SRC complete -0: 2026-10-07 15:50:59,366 INFO __main__ SRC's contraction-compression took 0.23186999500000002 s -0: 2026-10-07 15:50:59,711 INFO __main__ - Distance to reference: 1.4901161193847656e-08 (relative 1.4901161193847712e-08) -0: 2026-10-07 15:50:59,711 INFO __main__ benchmark_end -JobID JobName NCPUS AveCPU Elapsed ExitCode ------------- ---------- ---------- ---------- ---------- -------- -59626058 mpo_50_50+ 32 00:00:25 0:0 -59626058.ba+ batch 32 00:00:25 0:0 -59626058.ex+ extern 32 00:00:25 0:0 -59626058.0 python 32 00:00:21 0:0 -JobID JobName MaxRSS MaxRSSNode ------------- ---------- ---------- ---------- -59626058 mpo_50_50+ -59626058.ba+ batch -59626058.ex+ extern -59626058.0 python 0.57G lrdn1912 diff --git a/benches/primitives/leonardo/logs/mpo_50_50_src_gpu_59626787.out b/benches/primitives/leonardo/logs/mpo_50_50_src_gpu_59626787.out deleted file mode 100644 index 01d3916..0000000 --- a/benches/primitives/leonardo/logs/mpo_50_50_src_gpu_59626787.out +++ /dev/null @@ -1,42 +0,0 @@ -NODE: lrdn2682, CPUS: 32, ARGS: --n-sites 50 --chi-out 50 --device gpu -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -NVIDIA A100-SXM-64GB, 535.274.02 -0: 2026-10-07 16:02:04,119 INFO __main__ benchmark_start: n_sites=50, chi_out=50, chi_id=4, phys_dim=2, dtype=complex128, run=src, compare=yes, device=gpu -0: 2026-10-07 16:02:04,119 INFO __main__ Generating MPOs... -0: 2026-10-07 16:02:04,239 INFO __main__ Computing reference contraction (no compression)... -0: 2026-10-07 16:02:04,294 INFO __main__ Reference contraction took 0.054615671000000005 s -0: 2026-10-07 16:02:04,294 INFO __main__ Warming up the GPU... -0: 2026-10-07 16:02:26,535 DEBUG src_method.stack Starting SRC: n_sites=4, depth=2, output=mpo, device=cupy -0: 2026-10-07 16:02:27,853 DEBUG src_method._sweep SRC plan: prefetch=1, device peak=4224 B, host peak=2432 B, disk=0 B, scratch=None, tiers=['device', 'device', 'device', 'device'], batches (env, sketch, project)=[(2, 0, 2), (2, 2, 2), (2, 2, 2), (0, 2, 2)] -0: 2026-10-07 16:02:50,464 DEBUG src_method._sweep Left-to-right sweep: 22.354 s -0: 2026-10-07 16:03:00,804 DEBUG src_method._sweep Right-to-left sweep: 10.340 s -0: 2026-10-07 16:03:00,804 DEBUG src_method._sweep SRC stalls: sites 0.000 s, environments 0.000 s -0: 2026-10-07 16:03:00,804 DEBUG src_method._sweep Device pool: 48640 B -0: 2026-10-07 16:03:00,804 DEBUG src_method.stack SRC complete -0: 2026-10-07 16:03:00,804 INFO __main__ Computing SRC's MPO-MPO contraction (with compression)... -0: 2026-10-07 16:03:00,804 DEBUG src_method.stack Starting SRC: n_sites=50, depth=2, output=mpo, device=cupy -0: 2026-10-07 16:03:00,816 DEBUG src_method._sweep SRC plan: prefetch=1, device peak=9974272 B, host peak=8326272 B, disk=0 B, scratch=None, tiers=['device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device', 'device'], batches (env, sketch, project)=[(32, 0, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32 -0: ), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (32, 32, 32), (0, 32, 32)] -0: 2026-10-07 16:03:00,896 DEBUG src_method._sweep Left-to-right sweep: 0.080 s -0: 2026-10-07 16:03:01,376 DEBUG src_method._sweep Right-to-left sweep: 0.480 s -0: 2026-10-07 16:03:01,376 DEBUG src_method._sweep SRC stalls: sites 0.000 s, environments 0.000 s -0: 2026-10-07 16:03:01,376 DEBUG src_method._sweep Device pool: 11751424 B -0: 2026-10-07 16:03:01,376 DEBUG src_method.stack SRC complete -0: 2026-10-07 16:03:01,376 INFO __main__ SRC's contraction-compression took 0.572353006 s -0: 2026-10-07 16:03:01,376 INFO __main__ - CuPy pool high-water mark: 0.01 GB -0: 2026-10-07 16:03:01,721 INFO __main__ - Distance to reference: 1.4901161193847656e-08 (relative 1.4901161193847712e-08) -0: 2026-10-07 16:03:01,721 INFO __main__ benchmark_end -JobID JobName NCPUS AveCPU Elapsed ExitCode ------------- ---------- ---------- ---------- ---------- -------- -59626787 mpo_50_50+ 32 00:01:26 0:0 -59626787.ba+ batch 32 00:01:26 0:0 -59626787.ex+ extern 32 00:01:26 0:0 -59626787.0 python 32 00:01:22 0:0 -JobID JobName MaxRSS MaxRSSNode ------------- ---------- ---------- ---------- -59626787 mpo_50_50+ -59626787.ba+ batch -59626787.ex+ extern -59626787.0 python diff --git a/benches/primitives/leonardo/logs/mpo_mpo_1000_112cpu.out b/benches/primitives/leonardo/logs/mpo_mpo_1000_112cpu.out deleted file mode 100644 index b7afd44..0000000 --- a/benches/primitives/leonardo/logs/mpo_mpo_1000_112cpu.out +++ /dev/null @@ -1,120 +0,0 @@ -NODE: lrdn4451 -CPUS: 112 -srun: defined options -srun: -------------------- -------------------- -srun: (null) : lrdn4451 -srun: cpus-per-task : 112 -srun: jobid : 19685741 -srun: job-name : src_bench -srun: label : set -srun: mem : 440G -srun: nodes : 1 -srun: ntasks : 1 -srun: ntasks-per-node : 1 -srun: tres-per-task : cpu:112 -srun: verbose : 1 -srun: -------------------- -------------------- -srun: end of defined options -srun: jobid 19685741: nodes(1):`lrdn4451', cpu counts: 112(x1) -srun: Implicitly setting --exact, because -c/--cpus-per-task given. -srun: CpuBindType=none -srun: launching StepId=19685741.0 on host lrdn4451, 1 tasks: 0 -srun: topology/tree: init: topology tree plugin loaded -srun: Node lrdn4451, 1 tasks started -0: 2025-09-05 13:28:20 [info ] benchmark_start chi_out=1000 compare=no dtype=complex128 n_sites=50 phys_dim=2 run=src -0: 2025-09-05 13:28:20 [info ] Generating MPOs... -0: 2025-09-05 13:28:29 [info ] Computing SRC's MPO-MPO contraction (with compression)... -0: 2025-09-05 13:28:29 [info ] Starting SRC MPO-MPO n_sites=50 phys_down=2 phys_up=2 -0: 2025-09-05 13:28:29 [info ] Left-to-right sweep: Computing C tensors... -0: 2025-09-05 13:28:29 [debug ] Contraction path for C tensors: path=([(0, 2), (0, 1), (0, 1)], Complete contraction: abc,ade,bfdh,cghe->afg -0: Naive scaling: 8 -0: Optimized scaling: 6 -0: Naive FLOP count: 5.120e+11 -0: Optimized FLOP count: 3.213e+10 -0: Theoretical speedup: 1.594e+1 -0: Largest intermediate: 1.600e+7 elements -0: -------------------------------------------------------------------------------- -0: scaling BLAS current remaining -0: -------------------------------------------------------------------------------- -0: 6 TDOT bfdh,abc->fdhac ade,cghe,fdhac->afg -0: 6 GEMM cghe,ade->cghad fdhac,cghad->afg -0: 6 0 cghad,fdhac->afg afg->afg) -0: 2025-09-05 13:29:11 [debug ] - Step took 42.040742876 s -0: 2025-09-05 13:29:11 [info ] Right-to-left sweep: Constructing η tensors... -0: 2025-09-05 13:29:11 [debug ] Contraction path for η tensors: path=([(0, 1), (0, 2), (0, 1)], Complete contraction: abc,bdfg,cegh,ide->ifha -0: Naive scaling: 9 -0: Optimized scaling: 7 -0: Naive FLOP count: 5.120e+14 -0: Optimized FLOP count: 6.426e+10 -0: Theoretical speedup: 7.968e+3 -0: Largest intermediate: 1.600e+7 elements -0: -------------------------------------------------------------------------------- -0: scaling BLAS current remaining -0: -------------------------------------------------------------------------------- -0: 6 TDOT bdfg,abc->dfgac cegh,ide,dfgac->ifha -0: 7 TDOT dfgac,cegh->dfaeh ide,dfaeh->ifha -0: 6 TDOT dfaeh,ide->ifha ifha->ifha) -0: 2025-09-05 13:30:59 [debug ] Contraction path for S tensors: path=([(0, 3), (1, 2), (0, 1)], Complete contraction: abcd,hecg,ifgd,bef->ahi -0: Naive scaling: 9 -0: Optimized scaling: 7 -0: Naive FLOP count: 5.120e+14 -0: Optimized FLOP count: 6.426e+10 -0: Theoretical speedup: 7.968e+3 -0: Largest intermediate: 1.600e+7 elements -0: -------------------------------------------------------------------------------- -0: scaling BLAS current remaining -0: -------------------------------------------------------------------------------- -0: 6 TDOT bef,abcd->efacd hecg,ifgd,efacd->ahi -0: 7 TDOT efacd,ifgd->eacig hecg,eacig->ahi -0: 6 TDOT eacig,hecg->ahi ahi->ahi) -0: 2025-09-05 13:30:59 [debug ] - Step took 108.07037480800001 s -0: 2025-09-05 13:30:59 [info ] SRC MPO-MPO complete. -0: 2025-09-05 13:30:59 [info ] SRC's contraction-compression took 150.115254443 s -0: 2025-09-05 13:30:59 [info ] benchmark_end -srun: Received task exit notification for 1 task of StepId=19685741.0 (status=0x0000). -srun: lrdn4451: task 0: Completed - - -#------------------------# -Task and CPU usage stats: -JobID JobName NCPUS NNodes NTasks AveCPU MinCPU MinCPUNode MinCPUTask Elapsed ExitCode ------------- ---------- ---------- -------- -------- ---------- ---------- ---------- ---------- ---------- -------- -19685741 src_bench 112 1 00:03:21 0:0 -19685741.ba+ batch 112 1 1 00:03:21 0:0 -19685741.ex+ extern 112 1 1 00:03:21 0:0 -19685741.0 python 112 1 1 01:54:37 01:54:37 lrdn4451 0 00:03:19 0:0 -Memory usage stats: -JobID JobName AveRSS MaxRSS MaxRSSNode MaxRSSTask AvePages MaxPages MaxPagesNode MaxPagesTask ------------- ---------- -------------------------------------------------- -------------------------------------------------- ---------- ---------- ---------- -------- ------------ -------------- -19685741 src_bench -19685741.ba+ batch -19685741.ex+ extern -19685741.0 python 7.01G 7.01G lrdn4451 0 0.00G 0.00G lrdn4451 0 -Disk usage stats: -JobID JobName AveDiskRead MaxDiskRead MaxDiskReadNode MaxDiskReadTask AveDiskWrite MaxDiskWrite MaxDiskWriteNode MaxDiskWriteTask ------------- ---------- -------------- ------------ --------------- --------------- -------------- ------------ ---------------- ---------------- -19685741 src_bench -19685741.ba+ batch -19685741.ex+ extern -19685741.0 python 0.03G 0.03G lrdn4451 0 0.00M 0.00M lrdn4451 0 -Trackable resources usage stats: -JobID JobName AllocNodes AllocCPUS AllocTRES ------------- ---------- ---------- ---------- ---------------------------------------------------------------------------------------------------- -19685741 src_bench 1 112 billing=112,cpu=112,gres/tmpfs=3298534883328,mem=440G,node=1 -19685741.ba+ batch 1 112 cpu=112,gres/tmpfs=3298534883328,mem=440G,node=1 -19685741.ex+ extern 1 112 billing=112,cpu=112,gres/tmpfs=3298534883328,mem=440G,node=1 -19685741.0 python 1 112 cpu=112,gres/tmpfs=10737418240,mem=440G,node=1 -Trackable resources (ingress) usage stats: -JobID JobName TRESUsageInTot TRESUsageInAve TRESUsageInMax TRESUsageInMaxNode TRESUsageInMaxTask ------------- ---------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- -19685741 src_bench -19685741.ba+ batch -19685741.ex+ extern -19685741.0 python cpu=01:54:37,energy=0,fs/disk=0.03G,mem=7.01G,pages=0.00G,vmem=7.20G cpu=01:54:37,energy=0,fs/disk=0.03G,mem=7.01G,pages=0.00G,vmem=7.20G cpu=01:54:37,energy=0,fs/disk=0.03G,mem=7.01G,pages=0.00G,vmem=7.20G cpu=lrdn4451,energy=lrdn4451,fs/disk=lrdn4451,mem=lrdn4451,pages=lrdn+ cpu=0,fs/disk=0,mem=0,pages=0,vmem=0 -Trackable resources (egress) usage stats: -JobID JobName TRESUsageOutTot TRESUsageOutAve TRESUsageOutMax TRESUsageOutMaxNode TRESUsageOutMaxTask ------------- ---------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- -19685741 src_bench -19685741.ba+ batch -19685741.ex+ extern -19685741.0 python energy=0,fs/disk=0.00G energy=0,fs/disk=0.00G energy=0,fs/disk=0.00G energy=lrdn4451,fs/disk=lrdn4451 fs/disk=0 diff --git a/benches/primitives/leonardo/logs/mpo_mpo_1000_19685507.out b/benches/primitives/leonardo/logs/mpo_mpo_1000_19685507.out deleted file mode 100644 index 262de4b..0000000 --- a/benches/primitives/leonardo/logs/mpo_mpo_1000_19685507.out +++ /dev/null @@ -1,120 +0,0 @@ -NODE: lrdn3765 -CPUS: 112 -srun: defined options -srun: -------------------- -------------------- -srun: (null) : lrdn3765 -srun: cpus-per-task : 112 -srun: jobid : 19685507 -srun: job-name : src_bench -srun: label : set -srun: mem : 440G -srun: nodes : 1 -srun: ntasks : 1 -srun: ntasks-per-node : 1 -srun: tres-per-task : cpu:112 -srun: verbose : 1 -srun: -------------------- -------------------- -srun: end of defined options -srun: jobid 19685507: nodes(1):`lrdn3765', cpu counts: 112(x1) -srun: Implicitly setting --exact, because -c/--cpus-per-task given. -srun: CpuBindType=none -srun: launching StepId=19685507.0 on host lrdn3765, 1 tasks: 0 -srun: topology/tree: init: topology tree plugin loaded -srun: Node lrdn3765, 1 tasks started -0: 2025-09-05 13:16:24 [info ] benchmark_start chi_out=1000 compare=no dtype=complex128 n_sites=25 phys_dim=2 run=src -0: 2025-09-05 13:16:24 [info ] Generating MPOs... -0: 2025-09-05 13:16:28 [info ] Computing SRC's MPO-MPO contraction (with compression)... -0: 2025-09-05 13:16:28 [info ] Starting SRC MPO-MPO n_sites=25 phys_down=2 phys_up=2 -0: 2025-09-05 13:16:28 [info ] Left-to-right sweep: Computing C tensors... -0: 2025-09-05 13:16:28 [debug ] Contraction path for C tensors: path=([(0, 2), (0, 1), (0, 1)], Complete contraction: abc,ade,bfdh,cghe->afg -0: Naive scaling: 8 -0: Optimized scaling: 6 -0: Naive FLOP count: 5.120e+11 -0: Optimized FLOP count: 3.213e+10 -0: Theoretical speedup: 1.594e+1 -0: Largest intermediate: 1.600e+7 elements -0: -------------------------------------------------------------------------------- -0: scaling BLAS current remaining -0: -------------------------------------------------------------------------------- -0: 6 TDOT bfdh,abc->fdhac ade,cghe,fdhac->afg -0: 6 GEMM cghe,ade->cghad fdhac,cghad->afg -0: 6 0 cghad,fdhac->afg afg->afg) -0: 2025-09-05 13:16:48 [debug ] - Step took 20.147758949 s -0: 2025-09-05 13:16:48 [info ] Right-to-left sweep: Constructing η tensors... -0: 2025-09-05 13:16:48 [debug ] Contraction path for η tensors: path=([(0, 1), (0, 2), (0, 1)], Complete contraction: abc,bdfg,cegh,ide->ifha -0: Naive scaling: 9 -0: Optimized scaling: 7 -0: Naive FLOP count: 5.120e+14 -0: Optimized FLOP count: 6.426e+10 -0: Theoretical speedup: 7.968e+3 -0: Largest intermediate: 1.600e+7 elements -0: -------------------------------------------------------------------------------- -0: scaling BLAS current remaining -0: -------------------------------------------------------------------------------- -0: 6 TDOT bdfg,abc->dfgac cegh,ide,dfgac->ifha -0: 7 TDOT dfgac,cegh->dfaeh ide,dfaeh->ifha -0: 6 TDOT dfaeh,ide->ifha ifha->ifha) -0: 2025-09-05 13:17:39 [debug ] Contraction path for S tensors: path=([(0, 3), (1, 2), (0, 1)], Complete contraction: abcd,hecg,ifgd,bef->ahi -0: Naive scaling: 9 -0: Optimized scaling: 7 -0: Naive FLOP count: 5.120e+14 -0: Optimized FLOP count: 6.426e+10 -0: Theoretical speedup: 7.968e+3 -0: Largest intermediate: 1.600e+7 elements -0: -------------------------------------------------------------------------------- -0: scaling BLAS current remaining -0: -------------------------------------------------------------------------------- -0: 6 TDOT bef,abcd->efacd hecg,ifgd,efacd->ahi -0: 7 TDOT efacd,ifgd->eacig hecg,eacig->ahi -0: 6 TDOT eacig,hecg->ahi ahi->ahi) -0: 2025-09-05 13:17:39 [debug ] - Step took 50.950710502 s -0: 2025-09-05 13:17:39 [info ] SRC MPO-MPO complete. -0: 2025-09-05 13:17:39 [info ] SRC's contraction-compression took 71.102247426 s -0: 2025-09-05 13:17:39 [info ] benchmark_end -srun: Received task exit notification for 1 task of StepId=19685507.0 (status=0x0000). -srun: lrdn3765: task 0: Completed - - -#------------------------# -Task and CPU usage stats: -JobID JobName NCPUS NNodes NTasks AveCPU MinCPU MinCPUNode MinCPUTask Elapsed ExitCode ------------- ---------- ---------- -------- -------- ---------- ---------- ---------- ---------- ---------- -------- -19685507 src_bench 112 1 00:02:01 0:0 -19685507.ba+ batch 112 1 1 00:02:01 0:0 -19685507.ex+ extern 112 1 1 00:02:01 0:0 -19685507.0 python 112 1 1 00:54:17 00:54:17 lrdn3765 0 00:01:56 0:0 -Memory usage stats: -JobID JobName AveRSS MaxRSS MaxRSSNode MaxRSSTask AvePages MaxPages MaxPagesNode MaxPagesTask ------------- ---------- -------------------------------------------------- -------------------------------------------------- ---------- ---------- ---------- -------- ------------ -------------- -19685507 src_bench -19685507.ba+ batch -19685507.ex+ extern -19685507.0 python 3.66G 3.66G lrdn3765 0 0.00G 0.00G lrdn3765 0 -Disk usage stats: -JobID JobName AveDiskRead MaxDiskRead MaxDiskReadNode MaxDiskReadTask AveDiskWrite MaxDiskWrite MaxDiskWriteNode MaxDiskWriteTask ------------- ---------- -------------- ------------ --------------- --------------- -------------- ------------ ---------------- ---------------- -19685507 src_bench -19685507.ba+ batch -19685507.ex+ extern -19685507.0 python 0.03G 0.03G lrdn3765 0 0.00M 0.00M lrdn3765 0 -Trackable resources usage stats: -JobID JobName AllocNodes AllocCPUS AllocTRES ------------- ---------- ---------- ---------- ---------------------------------------------------------------------------------------------------- -19685507 src_bench 1 112 billing=112,cpu=112,gres/tmpfs=3298534883328,mem=440G,node=1 -19685507.ba+ batch 1 112 cpu=112,gres/tmpfs=3298534883328,mem=440G,node=1 -19685507.ex+ extern 1 112 billing=112,cpu=112,gres/tmpfs=3298534883328,mem=440G,node=1 -19685507.0 python 1 112 cpu=112,gres/tmpfs=10737418240,mem=440G,node=1 -Trackable resources (ingress) usage stats: -JobID JobName TRESUsageInTot TRESUsageInAve TRESUsageInMax TRESUsageInMaxNode TRESUsageInMaxTask ------------- ---------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- -19685507 src_bench -19685507.ba+ batch -19685507.ex+ extern -19685507.0 python cpu=00:54:17,energy=0,fs/disk=0.03G,mem=3.66G,pages=0.00G,vmem=3.84G cpu=00:54:17,energy=0,fs/disk=0.03G,mem=3.66G,pages=0.00G,vmem=3.84G cpu=00:54:17,energy=0,fs/disk=0.03G,mem=3.66G,pages=0.00G,vmem=3.84G cpu=lrdn3765,energy=lrdn3765,fs/disk=lrdn3765,mem=lrdn3765,pages=lrdn+ cpu=0,fs/disk=0,mem=0,pages=0,vmem=0 -Trackable resources (egress) usage stats: -JobID JobName TRESUsageOutTot TRESUsageOutAve TRESUsageOutMax TRESUsageOutMaxNode TRESUsageOutMaxTask ------------- ---------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- -19685507 src_bench -19685507.ba+ batch -19685507.ex+ extern -19685507.0 python energy=0,fs/disk=0.00G energy=0,fs/disk=0.00G energy=0,fs/disk=0.00G energy=lrdn3765,fs/disk=lrdn3765 fs/disk=0 diff --git a/benches/primitives/leonardo/logs/mpo_mpo_100_32cpu.out b/benches/primitives/leonardo/logs/mpo_mpo_100_32cpu.out deleted file mode 100644 index 7fb3333..0000000 --- a/benches/primitives/leonardo/logs/mpo_mpo_100_32cpu.out +++ /dev/null @@ -1,85 +0,0 @@ -Loading cuda/12.2 - Loading requirement: libiconv/1.17-nhc3mhm xz/5.4.6-xxxg42c - zlib-ng/2.1.6-jkgunjc libxml2/2.10.3-zbbe7lm - -Loading openmpi/4.1.6--gcc--12.2.0-cuda-12.2 - Loading requirement: glibc/2.28--gcc--12.2.0-gi6mmti - gcc-runtime/12.2.0--gcc--12.2.0-dqfwf7y numactl/2.0.14--gcc--12.2.0-cxzp4ci - bzip2/1.0.8-ib3znej ncurses/6.5-svfl57u pigz/2.8-5bwzpml zstd/1.5.6-uq5yyux - tar/1.34-jgektnv gettext/0.22.5-hsxgafg krb5/1.20.1-n4kmebh - libedit/3.1-20230828-nme7boz libxcrypt/4.4.35-7om46b5 openssh/9.7p1-tk2b5yv -NODE: lrdn0092 -CPUS: 32 -srun: defined options -srun: -------------------- -------------------- -srun: (null) : lrdn0092 -srun: cpus-per-task : 32 -srun: distribution : cyclic -srun: jobid : 19650980 -srun: job-name : src_bench -srun: label : set -srun: mem : 440G -srun: nodes : 1 -srun: ntasks : 1 -srun: ntasks-per-node : 1 -srun: tres-per-task : cpu:32 -srun: verbose : 1 -srun: -------------------- -------------------- -srun: end of defined options -srun: jobid 19650980: nodes(1):`lrdn0092', cpu counts: 32(x1) -srun: Implicitly setting --exact, because -c/--cpus-per-task given. -srun: CpuBindType=none -srun: launching StepId=19650980.0 on host lrdn0092, 1 tasks: 0 -srun: topology/tree: init: topology tree plugin loaded -srun: Node lrdn0092, 1 tasks started -0: INFO:__main__: Quimb's MPO-MPO contraction-compression took 2004.523261921 s -0: INFO:__main__: - Distance to reference: 0.0 -0: INFO:__main__: SRC's MPO-MPO contraction-compression took 94.82954199000001 s -0: INFO:__main__: - Distance to reference: 2.5809568279517847e-08 -srun: Received task exit notification for 1 task of StepId=19650980.0 (status=0x0000). -srun: lrdn0092: task 0: Completed - - -#------------------------# -Task and CPU usage stats: -JobID JobName NCPUS NNodes NTasks AveCPU MinCPU MinCPUNode MinCPUTask Elapsed ExitCode ------------- ---------- ---------- -------- -------- ---------- ---------- ---------- ---------- ---------- -------- -19650980 src_bench 32 1 00:57:44 0:0 -19650980.ba+ batch 32 1 1 00:57:44 0:0 -19650980.ex+ extern 32 1 1 00:57:44 0:0 -19650980.0 python 32 1 1 23:26:24 23:26:24 lrdn0092 0 00:57:39 0:0 -Memory usage stats: -JobID JobName AveRSS MaxRSS MaxRSSNode MaxRSSTask AvePages MaxPages MaxPagesNode MaxPagesTask ------------- ---------- -------------------------------------------------- -------------------------------------------------- ---------- ---------- ---------- -------- ------------ -------------- -19650980 src_bench -19650980.ba+ batch -19650980.ex+ extern -19650980.0 python 240.79G 240.79G lrdn0092 0 0.00G 0.00G lrdn0092 0 -Disk usage stats: -JobID JobName AveDiskRead MaxDiskRead MaxDiskReadNode MaxDiskReadTask AveDiskWrite MaxDiskWrite MaxDiskWriteNode MaxDiskWriteTask ------------- ---------- -------------- ------------ --------------- --------------- -------------- ------------ ---------------- ---------------- -19650980 src_bench -19650980.ba+ batch -19650980.ex+ extern -19650980.0 python 0.02G 0.02G lrdn0092 0 0.00M 0.00M lrdn0092 0 -Trackable resources usage stats: -JobID JobName AllocNodes AllocCPUS AllocTRES ------------- ---------- ---------- ---------- ---------------------------------------------------------------------------------------------------- -19650980 src_bench 1 32 billing=32,cpu=32,gres/gpu=4,mem=440G,node=1 -19650980.ba+ batch 1 32 cpu=32,gres/gpu=4,mem=440G,node=1 -19650980.ex+ extern 1 32 billing=32,cpu=32,gres/gpu=4,mem=440G,node=1 -19650980.0 python 1 32 cpu=32,mem=440G,node=1 -Trackable resources (ingress) usage stats: -JobID JobName TRESUsageInTot TRESUsageInAve TRESUsageInMax TRESUsageInMaxNode TRESUsageInMaxTask ------------- ---------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- -19650980 src_bench -19650980.ba+ batch -19650980.ex+ extern -19650980.0 python cpu=23:26:24,energy=0,fs/disk=0.02G,mem=240.79G,pages=0.00G,vmem=240.+ cpu=23:26:24,energy=0,fs/disk=0.02G,mem=240.79G,pages=0.00G,vmem=240.+ cpu=23:26:24,energy=0,fs/disk=0.02G,mem=240.79G,pages=0.00G,vmem=240.+ cpu=lrdn0092,energy=lrdn0092,fs/disk=lrdn0092,mem=lrdn0092,pages=lrdn+ cpu=0,fs/disk=0,mem=0,pages=0,vmem=0 -Trackable resources (egress) usage stats: -JobID JobName TRESUsageOutTot TRESUsageOutAve TRESUsageOutMax TRESUsageOutMaxNode TRESUsageOutMaxTask ------------- ---------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- -19650980 src_bench -19650980.ba+ batch -19650980.ex+ extern -19650980.0 python energy=0,fs/disk=0.00G energy=0,fs/disk=0.00G energy=0,fs/disk=0.00G energy=lrdn0092,fs/disk=lrdn0092 fs/disk=0 diff --git a/benches/primitives/leonardo/logs/mpo_mpo_50_112cpu.out b/benches/primitives/leonardo/logs/mpo_mpo_50_112cpu.out deleted file mode 100644 index b1d6f7e..0000000 --- a/benches/primitives/leonardo/logs/mpo_mpo_50_112cpu.out +++ /dev/null @@ -1,115 +0,0 @@ -NODE: lrdn4380 -CPUS: 112 -srun: defined options -srun: -------------------- -------------------- -srun: (null) : lrdn4380 -srun: cpus-per-task : 112 -srun: jobid : 19656379 -srun: job-name : src_bench -srun: label : set -srun: mem : 440G -srun: nodes : 1 -srun: ntasks : 1 -srun: ntasks-per-node : 1 -srun: tres-per-task : cpu:112 -srun: verbose : 1 -srun: -------------------- -------------------- -srun: end of defined options -srun: jobid 19656379: nodes(1):`lrdn4380', cpu counts: 112(x1) -srun: Implicitly setting --exact, because -c/--cpus-per-task given. -srun: CpuBindType=none -srun: launching StepId=19656379.0 on host lrdn4380, 1 tasks: 0 -srun: topology/tree: init: topology tree plugin loaded -srun: Node lrdn4380, 1 tasks started -0: 2025-09-04 13:19:06 [info ] benchmark_start chi_out=50 dtype=complex128 n_sites=50 phys_dim=2 run=quimb -0: 2025-09-04 13:19:06 [info ] Generating MPOs... -0: 2025-09-04 13:19:06 [info ] Computing reference contraction (no compression)... -0: 2025-09-04 13:19:26 [info ] Reference contraction took 19.310993082 s -0: 2025-09-04 13:19:26 [info ] Computing Quimb's MPO-MPO contraction (with compression)... -0: 2025-09-04 13:23:24 [info ] Quimb's contraction-compression took 238.66965269500002 s -0: 2025-09-04 13:24:06 [info ] - Distance to reference: 2.9802322387695312e-08 -0: 2025-09-04 13:24:06 [info ] benchmark_end -srun: Received task exit notification for 1 task of StepId=19656379.0 (status=0x0000). -srun: lrdn4380: task 0: Completed -srun: defined options -srun: -------------------- -------------------- -srun: (null) : lrdn4380 -srun: cpus-per-task : 112 -srun: jobid : 19656379 -srun: job-name : src_bench -srun: label : set -srun: mem : 440G -srun: nodes : 1 -srun: ntasks : 1 -srun: ntasks-per-node : 1 -srun: tres-per-task : cpu:112 -srun: verbose : 1 -srun: -------------------- -------------------- -srun: end of defined options -srun: jobid 19656379: nodes(1):`lrdn4380', cpu counts: 112(x1) -srun: Implicitly setting --exact, because -c/--cpus-per-task given. -srun: CpuBindType=none -srun: launching StepId=19656379.1 on host lrdn4380, 1 tasks: 0 -srun: topology/tree: init: topology tree plugin loaded -srun: Node lrdn4380, 1 tasks started -0: 2025-09-04 13:24:09 [info ] benchmark_start chi_out=50 dtype=complex128 n_sites=50 phys_dim=2 run=src -0: 2025-09-04 13:24:09 [info ] Generating MPOs... -0: 2025-09-04 13:24:10 [info ] Computing reference contraction (no compression)... -0: 2025-09-04 13:24:28 [info ] Reference contraction took 18.876444300000003 s -0: 2025-09-04 13:24:28 [info ] Computing SRC's MPO-MPO contraction (with compression)... -0: 2025-09-04 13:24:38 [info ] SRC's contraction-compression took 9.959594947000001 s -0: 2025-09-04 13:25:19 [info ] - Distance to reference: 2.9802322387695312e-08 -0: 2025-09-04 13:25:19 [info ] benchmark_end -srun: Received task exit notification for 1 task of StepId=19656379.1 (status=0x0000). -srun: lrdn4380: task 0: Completed - - -#------------------------# -Task and CPU usage stats: -JobID JobName NCPUS NNodes NTasks AveCPU MinCPU MinCPUNode MinCPUTask Elapsed ExitCode ------------- ---------- ---------- -------- -------- ---------- ---------- ---------- ---------- ---------- -------- -19656379 src_bench 112 1 00:07:04 0:0 -19656379.ba+ batch 112 1 1 00:07:04 0:0 -19656379.ex+ extern 112 1 1 00:07:04 0:0 -19656379.0 python 112 1 1 04:04:41 04:04:41 lrdn4380 0 00:05:48 0:0 -19656379.1 python 112 1 1 00:33:47 00:33:47 lrdn4380 0 00:01:13 0:0 -Memory usage stats: -JobID JobName AveRSS MaxRSS MaxRSSNode MaxRSSTask AvePages MaxPages MaxPagesNode MaxPagesTask ------------- ---------- -------------------------------------------------- -------------------------------------------------- ---------- ---------- ---------- -------- ------------ -------------- -19656379 src_bench -19656379.ba+ batch -19656379.ex+ extern -19656379.0 python 38.87G 38.87G lrdn4380 0 0.00G 0.00G lrdn4380 0 -19656379.1 python 38.33G 38.33G lrdn4380 0 0.00G 0.00G lrdn4380 0 -Disk usage stats: -JobID JobName AveDiskRead MaxDiskRead MaxDiskReadNode MaxDiskReadTask AveDiskWrite MaxDiskWrite MaxDiskWriteNode MaxDiskWriteTask ------------- ---------- -------------- ------------ --------------- --------------- -------------- ------------ ---------------- ---------------- -19656379 src_bench -19656379.ba+ batch -19656379.ex+ extern -19656379.0 python 0.03G 0.03G lrdn4380 0 0.00G 0.00G lrdn4380 0 -19656379.1 python 0.03G 0.03G lrdn4380 0 0.00M 0.00M lrdn4380 0 -Trackable resources usage stats: -JobID JobName AllocNodes AllocCPUS AllocTRES ------------- ---------- ---------- ---------- ---------------------------------------------------------------------------------------------------- -19656379 src_bench 1 112 billing=112,cpu=112,gres/tmpfs=3298534883328,mem=440G,node=1 -19656379.ba+ batch 1 112 cpu=112,gres/tmpfs=3298534883328,mem=440G,node=1 -19656379.ex+ extern 1 112 billing=112,cpu=112,gres/tmpfs=3298534883328,mem=440G,node=1 -19656379.0 python 1 112 cpu=112,gres/tmpfs=10737418240,mem=440G,node=1 -19656379.1 python 1 112 cpu=112,gres/tmpfs=10737418240,mem=440G,node=1 -Trackable resources (ingress) usage stats: -JobID JobName TRESUsageInTot TRESUsageInAve TRESUsageInMax TRESUsageInMaxNode TRESUsageInMaxTask ------------- ---------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- -19656379 src_bench -19656379.ba+ batch -19656379.ex+ extern -19656379.0 python cpu=04:04:41,energy=0,fs/disk=0.03G,mem=38.87G,pages=0.00G,vmem=39.06G cpu=04:04:41,energy=0,fs/disk=0.03G,mem=38.87G,pages=0.00G,vmem=39.06G cpu=04:04:41,energy=0,fs/disk=0.03G,mem=38.87G,pages=0.00G,vmem=39.06G cpu=lrdn4380,energy=lrdn4380,fs/disk=lrdn4380,mem=lrdn4380,pages=lrdn+ cpu=0,fs/disk=0,mem=0,pages=0,vmem=0 -19656379.1 python cpu=00:33:47,energy=0,fs/disk=0.03G,mem=38.33G,pages=0.00G,vmem=38.33G cpu=00:33:47,energy=0,fs/disk=0.03G,mem=38.33G,pages=0.00G,vmem=38.33G cpu=00:33:47,energy=0,fs/disk=0.03G,mem=38.33G,pages=0.00G,vmem=38.33G cpu=lrdn4380,energy=lrdn4380,fs/disk=lrdn4380,mem=lrdn4380,pages=lrdn+ cpu=0,fs/disk=0,mem=0,pages=0,vmem=0 -Trackable resources (egress) usage stats: -JobID JobName TRESUsageOutTot TRESUsageOutAve TRESUsageOutMax TRESUsageOutMaxNode TRESUsageOutMaxTask ------------- ---------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- -19656379 src_bench -19656379.ba+ batch -19656379.ex+ extern -19656379.0 python energy=0,fs/disk=0.00G energy=0,fs/disk=0.00G energy=0,fs/disk=0.00G energy=lrdn4380,fs/disk=lrdn4380 fs/disk=0 -19656379.1 python energy=0,fs/disk=0.00G energy=0,fs/disk=0.00G energy=0,fs/disk=0.00G energy=lrdn4380,fs/disk=lrdn4380 fs/disk=0 diff --git a/benches/primitives/leonardo/logs/mpo_mpo_50_32cpu.out b/benches/primitives/leonardo/logs/mpo_mpo_50_32cpu.out deleted file mode 100644 index 7d58a1a..0000000 --- a/benches/primitives/leonardo/logs/mpo_mpo_50_32cpu.out +++ /dev/null @@ -1,85 +0,0 @@ -Loading cuda/12.2 - Loading requirement: libiconv/1.17-nhc3mhm xz/5.4.6-xxxg42c - zlib-ng/2.1.6-jkgunjc libxml2/2.10.3-zbbe7lm - -Loading openmpi/4.1.6--gcc--12.2.0-cuda-12.2 - Loading requirement: glibc/2.28--gcc--12.2.0-gi6mmti - gcc-runtime/12.2.0--gcc--12.2.0-dqfwf7y numactl/2.0.14--gcc--12.2.0-cxzp4ci - bzip2/1.0.8-ib3znej ncurses/6.5-svfl57u pigz/2.8-5bwzpml zstd/1.5.6-uq5yyux - tar/1.34-jgektnv gettext/0.22.5-hsxgafg krb5/1.20.1-n4kmebh - libedit/3.1-20230828-nme7boz libxcrypt/4.4.35-7om46b5 openssh/9.7p1-tk2b5yv -NODE: lrdn0127 -CPUS: 32 -srun: defined options -srun: -------------------- -------------------- -srun: (null) : lrdn0127 -srun: cpus-per-task : 32 -srun: distribution : cyclic -srun: jobid : 19650306 -srun: job-name : src_bench -srun: label : set -srun: mem : 440G -srun: nodes : 1 -srun: ntasks : 1 -srun: ntasks-per-node : 1 -srun: tres-per-task : cpu:32 -srun: verbose : 1 -srun: -------------------- -------------------- -srun: end of defined options -srun: jobid 19650306: nodes(1):`lrdn0127', cpu counts: 32(x1) -srun: Implicitly setting --exact, because -c/--cpus-per-task given. -srun: CpuBindType=none -srun: launching StepId=19650306.0 on host lrdn0127, 1 tasks: 0 -srun: topology/tree: init: topology tree plugin loaded -srun: Node lrdn0127, 1 tasks started -0: INFO:__main__: Quimb's MPO-MPO contraction-compression took 164.970107243 s -0: INFO:__main__: - Distance to reference: 0.0 -0: INFO:__main__: SRC's MPO-MPO contraction-compression took 11.924697651 s -0: INFO:__main__: - Distance to reference: 2.1073424255447017e-08 -srun: Received task exit notification for 1 task of StepId=19650306.0 (status=0x0000). -srun: lrdn0127: task 0: Completed - - -#------------------------# -Task and CPU usage stats: -JobID JobName NCPUS NNodes NTasks AveCPU MinCPU MinCPUNode MinCPUTask Elapsed ExitCode ------------- ---------- ---------- -------- -------- ---------- ---------- ---------- ---------- ---------- -------- -19650306 src_bench 32 1 00:06:01 0:0 -19650306.ba+ batch 32 1 1 00:06:01 0:0 -19650306.ex+ extern 32 1 1 00:06:01 0:0 -19650306.0 python 32 1 1 01:52:59 01:52:59 lrdn0127 0 00:05:56 0:0 -Memory usage stats: -JobID JobName AveRSS MaxRSS MaxRSSNode MaxRSSTask AvePages MaxPages MaxPagesNode MaxPagesTask ------------- ---------- -------------------------------------------------- -------------------------------------------------- ---------- ---------- ---------- -------- ------------ -------------- -19650306 src_bench -19650306.ba+ batch -19650306.ex+ extern -19650306.0 python 38.79G 38.79G lrdn0127 0 0.00G 0.00G lrdn0127 0 -Disk usage stats: -JobID JobName AveDiskRead MaxDiskRead MaxDiskReadNode MaxDiskReadTask AveDiskWrite MaxDiskWrite MaxDiskWriteNode MaxDiskWriteTask ------------- ---------- -------------- ------------ --------------- --------------- -------------- ------------ ---------------- ---------------- -19650306 src_bench -19650306.ba+ batch -19650306.ex+ extern -19650306.0 python 0.02G 0.02G lrdn0127 0 0.00M 0.00M lrdn0127 0 -Trackable resources usage stats: -JobID JobName AllocNodes AllocCPUS AllocTRES ------------- ---------- ---------- ---------- ---------------------------------------------------------------------------------------------------- -19650306 src_bench 1 32 billing=32,cpu=32,gres/gpu=4,mem=440G,node=1 -19650306.ba+ batch 1 32 cpu=32,gres/gpu=4,mem=440G,node=1 -19650306.ex+ extern 1 32 billing=32,cpu=32,gres/gpu=4,mem=440G,node=1 -19650306.0 python 1 32 cpu=32,mem=440G,node=1 -Trackable resources (ingress) usage stats: -JobID JobName TRESUsageInTot TRESUsageInAve TRESUsageInMax TRESUsageInMaxNode TRESUsageInMaxTask ------------- ---------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- -19650306 src_bench -19650306.ba+ batch -19650306.ex+ extern -19650306.0 python cpu=01:52:59,energy=0,fs/disk=0.02G,mem=38.79G,pages=0.00G,vmem=38.98G cpu=01:52:59,energy=0,fs/disk=0.02G,mem=38.79G,pages=0.00G,vmem=38.98G cpu=01:52:59,energy=0,fs/disk=0.02G,mem=38.79G,pages=0.00G,vmem=38.98G cpu=lrdn0127,energy=lrdn0127,fs/disk=lrdn0127,mem=lrdn0127,pages=lrdn+ cpu=0,fs/disk=0,mem=0,pages=0,vmem=0 -Trackable resources (egress) usage stats: -JobID JobName TRESUsageOutTot TRESUsageOutAve TRESUsageOutMax TRESUsageOutMaxNode TRESUsageOutMaxTask ------------- ---------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- -19650306 src_bench -19650306.ba+ batch -19650306.ex+ extern -19650306.0 python energy=0,fs/disk=0.00G energy=0,fs/disk=0.00G energy=0,fs/disk=0.00G energy=lrdn0127,fs/disk=lrdn0127 fs/disk=0 diff --git a/benches/primitives/leonardo/run.sh b/benches/primitives/leonardo/run.sh deleted file mode 100755 index e5d46ea..0000000 --- a/benches/primitives/leonardo/run.sh +++ /dev/null @@ -1,38 +0,0 @@ -#!/usr/bin/env -S bash -l -# Usage, from this folder: sbatch run.sh [bench_mpo_mpo.py options] -# e.g. sbatch run.sh --n-sites 25 --chi-out 1000 --compare no --device gpu -# Add --qos=boost_qos_dbg (30 min, 2 nodes) to sbatch for short test runs. -#SBATCH --nodes=1 -#SBATCH --ntasks-per-node=1 -#SBATCH --job-name=mpo_mpo -#SBATCH --account=EUHPC_D30_139 -#SBATCH --partition=boost_usr_prod -#SBATCH --time=02:00:00 -#SBATCH --cpus-per-task=32 -#SBATCH --gres=gpu:1 -#SBATCH --mem=0 -#SBATCH --exclusive -#SBATCH --output=logs/%x_%j.out - -set -euo pipefail - -export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK -export OMP_PLACES=cores -export OMP_PROC_BIND=spread - -REPO=$(git -C "$SLURM_SUBMIT_DIR" rev-parse --show-toplevel) -source "$REPO/.venv/bin/activate" -export CUPY_CACHE_DIR=$REPO/.cupy-cache -# The Booster driver (535) predates CUDA 13: load NVIDIA's forward-compat libcuda. -CUDA_COMPAT=${CUDA_COMPAT:-$REPO/.cuda-compat/usr/local/cuda-13.4/compat} -if [[ -d $CUDA_COMPAT ]]; then - export LD_LIBRARY_PATH=$CUDA_COMPAT${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH} -fi - -echo "NODE: $SLURMD_NODENAME, CPUS: $SLURM_CPUS_PER_TASK, ARGS: $*" -nvidia-smi --query-gpu=name,driver_version --format=csv,noheader - -srun --label python bench_mpo_mpo.py "$@" - -sacct --format=JobID,JobName,NCPUS,AveCPU,Elapsed,ExitCode --jobs="$SLURM_JOB_ID" -sacct --format=JobID,JobName,MaxRSS,MaxRSSNode --units=G --jobs="$SLURM_JOB_ID" diff --git a/benches/stack/README.md b/benches/stack/README.md index 5c597f5..144ac9a 100644 --- a/benches/stack/README.md +++ b/benches/stack/README.md @@ -40,13 +40,13 @@ steps per sweep (`k = 1` is pairwise application), on one A100-64GB. The reference is `k = 1` at `chi = 1024`; rows give the wall time of the 80 steps, the infidelity of the final state and the error of `` on the middle qubit. `quimb` applies the same MPO with exact SVD truncation on the 32 cores of the -node. The script and logs are in [`leonardo/`](leonardo/); see the -[Leonardo section](../README.md#leonardo) for the environment. +node. See [HPC clusters](https://algorithmiq.github.io/src-method/hpc/leonardo) +for how to run on Leonardo. ```bash -sbatch run.sh evolve --n-sites 50 --quimb-max-chi 0 -sbatch run.sh evolve --n-sites 100 --quimb-max-chi 0 -sbatch run.sh evolve --n-sites 50 --chis 64 --quimb-max-chi 64 +uv run python benches/stack/bench_depth.py evolve --n-sites 50 --quimb-max-chi 0 +uv run python benches/stack/bench_depth.py evolve --n-sites 100 --quimb-max-chi 0 +uv run python benches/stack/bench_depth.py evolve --n-sites 50 --chis 64 --quimb-max-chi 64 ``` | qubits | chi | method | seconds | infidelity | z_error | diff --git a/benches/stack/leonardo/logs/evolve_100_src_59696296.out b/benches/stack/leonardo/logs/evolve_100_src_59696296.out deleted file mode 100644 index 086dac8..0000000 --- a/benches/stack/leonardo/logs/evolve_100_src_59696296.out +++ /dev/null @@ -1,26 +0,0 @@ -NODE: lrdn3359, CPUS: 32, ARGS: evolve --n-sites 100 --quimb-max-chi 0 -reference: chi=1024, 308.4 s, =0.667891 -done: {'n_sites': 100, 'chi': 64, 'method': 'src gpu k=1', 'seconds': 13.123708908446133, 'infidelity': 0.13562671056129083, 'z_error': 0.012618184169088664} -done: {'n_sites': 100, 'chi': 64, 'method': 'src gpu k=2', 'seconds': 7.625959983095527, 'infidelity': 0.1366349218135129, 'z_error': 0.015122156350154792} -done: {'n_sites': 100, 'chi': 64, 'method': 'src gpu k=4', 'seconds': 5.22486006654799, 'infidelity': 0.13816995945577903, 'z_error': 0.014127674647360755} -done: {'n_sites': 100, 'chi': 128, 'method': 'src gpu k=1', 'seconds': 15.661909165792167, 'infidelity': 0.04989325548928303, 'z_error': 0.005920579475230836} -done: {'n_sites': 100, 'chi': 128, 'method': 'src gpu k=2', 'seconds': 8.773102682083845, 'infidelity': 0.05092744315540787, 'z_error': 0.006398354943719209} -done: {'n_sites': 100, 'chi': 128, 'method': 'src gpu k=4', 'seconds': 5.613364351913333, 'infidelity': 0.051205208196916985, 'z_error': 0.006251898917556509} -done: {'n_sites': 100, 'chi': 256, 'method': 'src gpu k=1', 'seconds': 28.242494146339595, 'infidelity': 0.01312509062975109, 'z_error': 0.001679279815550938} -done: {'n_sites': 100, 'chi': 256, 'method': 'src gpu k=2', 'seconds': 15.410599228926003, 'infidelity': 0.013445609469341702, 'z_error': 0.0016643548594549307} -done: {'n_sites': 100, 'chi': 256, 'method': 'src gpu k=4', 'seconds': 11.582731399685144, 'infidelity': 0.013731984082151039, 'z_error': 0.001786955239251764} -done: {'n_sites': 100, 'chi': 512, 'method': 'src gpu k=1', 'seconds': 72.27939910441637, 'infidelity': 0.0017523314758546293, 'z_error': 0.00024527354217129815} -done: {'n_sites': 100, 'chi': 512, 'method': 'src gpu k=2', 'seconds': 40.612829772755504, 'infidelity': 0.0018407681059465775, 'z_error': 0.0002543728000593548} -done: {'n_sites': 100, 'chi': 512, 'method': 'src gpu k=4', 'seconds': 41.36851749941707, 'infidelity': 0.0018824815909065373, 'z_error': 0.0002470543323899177} -result: {'n_sites': 100, 'chi': 64, 'method': 'src gpu k=1', 'seconds': 13.123708908446133, 'infidelity': 0.13562671056129083, 'z_error': 0.012618184169088664} -result: {'n_sites': 100, 'chi': 64, 'method': 'src gpu k=2', 'seconds': 7.625959983095527, 'infidelity': 0.1366349218135129, 'z_error': 0.015122156350154792} -result: {'n_sites': 100, 'chi': 64, 'method': 'src gpu k=4', 'seconds': 5.22486006654799, 'infidelity': 0.13816995945577903, 'z_error': 0.014127674647360755} -result: {'n_sites': 100, 'chi': 128, 'method': 'src gpu k=1', 'seconds': 15.661909165792167, 'infidelity': 0.04989325548928303, 'z_error': 0.005920579475230836} -result: {'n_sites': 100, 'chi': 128, 'method': 'src gpu k=2', 'seconds': 8.773102682083845, 'infidelity': 0.05092744315540787, 'z_error': 0.006398354943719209} -result: {'n_sites': 100, 'chi': 128, 'method': 'src gpu k=4', 'seconds': 5.613364351913333, 'infidelity': 0.051205208196916985, 'z_error': 0.006251898917556509} -result: {'n_sites': 100, 'chi': 256, 'method': 'src gpu k=1', 'seconds': 28.242494146339595, 'infidelity': 0.01312509062975109, 'z_error': 0.001679279815550938} -result: {'n_sites': 100, 'chi': 256, 'method': 'src gpu k=2', 'seconds': 15.410599228926003, 'infidelity': 0.013445609469341702, 'z_error': 0.0016643548594549307} -result: {'n_sites': 100, 'chi': 256, 'method': 'src gpu k=4', 'seconds': 11.582731399685144, 'infidelity': 0.013731984082151039, 'z_error': 0.001786955239251764} -result: {'n_sites': 100, 'chi': 512, 'method': 'src gpu k=1', 'seconds': 72.27939910441637, 'infidelity': 0.0017523314758546293, 'z_error': 0.00024527354217129815} -result: {'n_sites': 100, 'chi': 512, 'method': 'src gpu k=2', 'seconds': 40.612829772755504, 'infidelity': 0.0018407681059465775, 'z_error': 0.0002543728000593548} -result: {'n_sites': 100, 'chi': 512, 'method': 'src gpu k=4', 'seconds': 41.36851749941707, 'infidelity': 0.0018824815909065373, 'z_error': 0.0002470543323899177} diff --git a/benches/stack/leonardo/logs/evolve_50_quimb_59699027.out b/benches/stack/leonardo/logs/evolve_50_quimb_59699027.out deleted file mode 100644 index 823e3f5..0000000 --- a/benches/stack/leonardo/logs/evolve_50_quimb_59699027.out +++ /dev/null @@ -1,10 +0,0 @@ -NODE: lrdn0954, CPUS: 32, ARGS: evolve --n-sites 50 --chis 64 --quimb-max-chi 64 -reference: chi=1024, 137.5 s, =0.667890 -done: {'n_sites': 50, 'chi': 64, 'method': 'src gpu k=1', 'seconds': 6.8661597250029445, 'infidelity': 0.06155440465779205, 'z_error': 0.01341688902307503} -done: {'n_sites': 50, 'chi': 64, 'method': 'src gpu k=2', 'seconds': 4.19635803392157, 'infidelity': 0.062078732788092506, 'z_error': 0.012000585371298333} -done: {'n_sites': 50, 'chi': 64, 'method': 'src gpu k=4', 'seconds': 2.8887054407969117, 'infidelity': 0.06313201831236248, 'z_error': 0.01346690615117696} -done: {'n_sites': 50, 'chi': 64, 'method': 'quimb cpu', 'seconds': 730.9863200872205, 'infidelity': 0.03682896328621432, 'z_error': 0.004079586034429261} -result: {'n_sites': 50, 'chi': 64, 'method': 'src gpu k=1', 'seconds': 6.8661597250029445, 'infidelity': 0.06155440465779205, 'z_error': 0.01341688902307503} -result: {'n_sites': 50, 'chi': 64, 'method': 'src gpu k=2', 'seconds': 4.19635803392157, 'infidelity': 0.062078732788092506, 'z_error': 0.012000585371298333} -result: {'n_sites': 50, 'chi': 64, 'method': 'src gpu k=4', 'seconds': 2.8887054407969117, 'infidelity': 0.06313201831236248, 'z_error': 0.01346690615117696} -result: {'n_sites': 50, 'chi': 64, 'method': 'quimb cpu', 'seconds': 730.9863200872205, 'infidelity': 0.03682896328621432, 'z_error': 0.004079586034429261} diff --git a/benches/stack/leonardo/logs/evolve_50_src_59696294.out b/benches/stack/leonardo/logs/evolve_50_src_59696294.out deleted file mode 100644 index 164f595..0000000 --- a/benches/stack/leonardo/logs/evolve_50_src_59696294.out +++ /dev/null @@ -1,26 +0,0 @@ -NODE: lrdn3333, CPUS: 32, ARGS: evolve --n-sites 50 --quimb-max-chi 0 -reference: chi=1024, 139.7 s, =0.667890 -done: {'n_sites': 50, 'chi': 64, 'method': 'src gpu k=1', 'seconds': 6.860480501316488, 'infidelity': 0.06155440465779205, 'z_error': 0.01341688902307503} -done: {'n_sites': 50, 'chi': 64, 'method': 'src gpu k=2', 'seconds': 4.161895754747093, 'infidelity': 0.062078732788092506, 'z_error': 0.012000585371298333} -done: {'n_sites': 50, 'chi': 64, 'method': 'src gpu k=4', 'seconds': 2.897886755876243, 'infidelity': 0.06313201831236248, 'z_error': 0.01346690615117696} -done: {'n_sites': 50, 'chi': 128, 'method': 'src gpu k=1', 'seconds': 7.610168346203864, 'infidelity': 0.021156790935796144, 'z_error': 0.0055355706430136875} -done: {'n_sites': 50, 'chi': 128, 'method': 'src gpu k=2', 'seconds': 4.489643581211567, 'infidelity': 0.021602470774183202, 'z_error': 0.0059963821739627} -done: {'n_sites': 50, 'chi': 128, 'method': 'src gpu k=4', 'seconds': 3.013490010984242, 'infidelity': 0.021751406122760164, 'z_error': 0.005964338019787685} -done: {'n_sites': 50, 'chi': 256, 'method': 'src gpu k=1', 'seconds': 13.911777798086405, 'infidelity': 0.005270284570452866, 'z_error': 0.0016805209553995004} -done: {'n_sites': 50, 'chi': 256, 'method': 'src gpu k=2', 'seconds': 7.818938770331442, 'infidelity': 0.005383701475480507, 'z_error': 0.0019074516085897386} -done: {'n_sites': 50, 'chi': 256, 'method': 'src gpu k=4', 'seconds': 5.861380219459534, 'infidelity': 0.005502160126604205, 'z_error': 0.0018748967672316263} -done: {'n_sites': 50, 'chi': 512, 'method': 'src gpu k=1', 'seconds': 33.87516096793115, 'infidelity': 0.0006622772550057832, 'z_error': 0.00023484147693342283} -done: {'n_sites': 50, 'chi': 512, 'method': 'src gpu k=2', 'seconds': 19.16721878014505, 'infidelity': 0.0006922507941875544, 'z_error': 0.00025889370930176625} -done: {'n_sites': 50, 'chi': 512, 'method': 'src gpu k=4', 'seconds': 19.139042486436665, 'infidelity': 0.0007114537093736439, 'z_error': 0.00027603449727797713} -result: {'n_sites': 50, 'chi': 64, 'method': 'src gpu k=1', 'seconds': 6.860480501316488, 'infidelity': 0.06155440465779205, 'z_error': 0.01341688902307503} -result: {'n_sites': 50, 'chi': 64, 'method': 'src gpu k=2', 'seconds': 4.161895754747093, 'infidelity': 0.062078732788092506, 'z_error': 0.012000585371298333} -result: {'n_sites': 50, 'chi': 64, 'method': 'src gpu k=4', 'seconds': 2.897886755876243, 'infidelity': 0.06313201831236248, 'z_error': 0.01346690615117696} -result: {'n_sites': 50, 'chi': 128, 'method': 'src gpu k=1', 'seconds': 7.610168346203864, 'infidelity': 0.021156790935796144, 'z_error': 0.0055355706430136875} -result: {'n_sites': 50, 'chi': 128, 'method': 'src gpu k=2', 'seconds': 4.489643581211567, 'infidelity': 0.021602470774183202, 'z_error': 0.0059963821739627} -result: {'n_sites': 50, 'chi': 128, 'method': 'src gpu k=4', 'seconds': 3.013490010984242, 'infidelity': 0.021751406122760164, 'z_error': 0.005964338019787685} -result: {'n_sites': 50, 'chi': 256, 'method': 'src gpu k=1', 'seconds': 13.911777798086405, 'infidelity': 0.005270284570452866, 'z_error': 0.0016805209553995004} -result: {'n_sites': 50, 'chi': 256, 'method': 'src gpu k=2', 'seconds': 7.818938770331442, 'infidelity': 0.005383701475480507, 'z_error': 0.0019074516085897386} -result: {'n_sites': 50, 'chi': 256, 'method': 'src gpu k=4', 'seconds': 5.861380219459534, 'infidelity': 0.005502160126604205, 'z_error': 0.0018748967672316263} -result: {'n_sites': 50, 'chi': 512, 'method': 'src gpu k=1', 'seconds': 33.87516096793115, 'infidelity': 0.0006622772550057832, 'z_error': 0.00023484147693342283} -result: {'n_sites': 50, 'chi': 512, 'method': 'src gpu k=2', 'seconds': 19.16721878014505, 'infidelity': 0.0006922507941875544, 'z_error': 0.00025889370930176625} -result: {'n_sites': 50, 'chi': 512, 'method': 'src gpu k=4', 'seconds': 19.139042486436665, 'infidelity': 0.0007114537093736439, 'z_error': 0.00027603449727797713} diff --git a/benches/stack/leonardo/logs/evolve_59696294.md b/benches/stack/leonardo/logs/evolve_59696294.md deleted file mode 100644 index 93b2f77..0000000 --- a/benches/stack/leonardo/logs/evolve_59696294.md +++ /dev/null @@ -1,14 +0,0 @@ -| n_sites | chi | method | seconds | infidelity | z_error | -|---|---|---|---|---|---| -| 50 | 64 | src gpu k=1 | 6.86 | 0.0616 | 0.0134 | -| 50 | 64 | src gpu k=2 | 4.16 | 0.0621 | 0.012 | -| 50 | 64 | src gpu k=4 | 2.9 | 0.0631 | 0.0135 | -| 50 | 128 | src gpu k=1 | 7.61 | 0.0212 | 0.00554 | -| 50 | 128 | src gpu k=2 | 4.49 | 0.0216 | 0.006 | -| 50 | 128 | src gpu k=4 | 3.01 | 0.0218 | 0.00596 | -| 50 | 256 | src gpu k=1 | 13.9 | 0.00527 | 0.00168 | -| 50 | 256 | src gpu k=2 | 7.82 | 0.00538 | 0.00191 | -| 50 | 256 | src gpu k=4 | 5.86 | 0.0055 | 0.00187 | -| 50 | 512 | src gpu k=1 | 33.9 | 0.000662 | 0.000235 | -| 50 | 512 | src gpu k=2 | 19.2 | 0.000692 | 0.000259 | -| 50 | 512 | src gpu k=4 | 19.1 | 0.000711 | 0.000276 | diff --git a/benches/stack/leonardo/logs/evolve_59696296.md b/benches/stack/leonardo/logs/evolve_59696296.md deleted file mode 100644 index c863d9c..0000000 --- a/benches/stack/leonardo/logs/evolve_59696296.md +++ /dev/null @@ -1,14 +0,0 @@ -| n_sites | chi | method | seconds | infidelity | z_error | -|---|---|---|---|---|---| -| 100 | 64 | src gpu k=1 | 13.1 | 0.136 | 0.0126 | -| 100 | 64 | src gpu k=2 | 7.63 | 0.137 | 0.0151 | -| 100 | 64 | src gpu k=4 | 5.22 | 0.138 | 0.0141 | -| 100 | 128 | src gpu k=1 | 15.7 | 0.0499 | 0.00592 | -| 100 | 128 | src gpu k=2 | 8.77 | 0.0509 | 0.0064 | -| 100 | 128 | src gpu k=4 | 5.61 | 0.0512 | 0.00625 | -| 100 | 256 | src gpu k=1 | 28.2 | 0.0131 | 0.00168 | -| 100 | 256 | src gpu k=2 | 15.4 | 0.0134 | 0.00166 | -| 100 | 256 | src gpu k=4 | 11.6 | 0.0137 | 0.00179 | -| 100 | 512 | src gpu k=1 | 72.3 | 0.00175 | 0.000245 | -| 100 | 512 | src gpu k=2 | 40.6 | 0.00184 | 0.000254 | -| 100 | 512 | src gpu k=4 | 41.4 | 0.00188 | 0.000247 | diff --git a/benches/stack/leonardo/logs/evolve_59699027.md b/benches/stack/leonardo/logs/evolve_59699027.md deleted file mode 100644 index 69205c3..0000000 --- a/benches/stack/leonardo/logs/evolve_59699027.md +++ /dev/null @@ -1,6 +0,0 @@ -| n_sites | chi | method | seconds | infidelity | z_error | -|---|---|---|---|---|---| -| 50 | 64 | src gpu k=1 | 6.87 | 0.0616 | 0.0134 | -| 50 | 64 | src gpu k=2 | 4.2 | 0.0621 | 0.012 | -| 50 | 64 | src gpu k=4 | 2.89 | 0.0631 | 0.0135 | -| 50 | 64 | quimb cpu | 731 | 0.0368 | 0.00408 | diff --git a/benches/stack/leonardo/run.sh b/benches/stack/leonardo/run.sh deleted file mode 100755 index 785058d..0000000 --- a/benches/stack/leonardo/run.sh +++ /dev/null @@ -1,34 +0,0 @@ -#!/usr/bin/env -S bash -l -# Usage, from this folder: sbatch run.sh evolve [bench_depth.py evolve options] -# accuracy and timing are laptop-sized; run them locally. -# Add --qos=boost_qos_dbg (30 min, 2 nodes) to sbatch for short test runs. -#SBATCH --nodes=1 -#SBATCH --ntasks-per-node=1 -#SBATCH --job-name=stack -#SBATCH --account=EUHPC_D30_139 -#SBATCH --partition=boost_usr_prod -#SBATCH --time=02:00:00 -#SBATCH --cpus-per-task=32 -#SBATCH --gres=gpu:1 -#SBATCH --mem=0 -#SBATCH --exclusive -#SBATCH --output=logs/%x_%j.out - -set -euo pipefail - -export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK -export OMP_PLACES=cores -export OMP_PROC_BIND=spread - -REPO=$(git -C "$SLURM_SUBMIT_DIR" rev-parse --show-toplevel) -source "$REPO/.venv/bin/activate" -export CUPY_CACHE_DIR=$REPO/.cupy-cache -# The Booster driver (535) predates CUDA 13: load NVIDIA's forward-compat libcuda. -CUDA_COMPAT=${CUDA_COMPAT:-$REPO/.cuda-compat/usr/local/cuda-13.4/compat} -if [[ -d $CUDA_COMPAT ]]; then - export LD_LIBRARY_PATH=$CUDA_COMPAT${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH} -fi - -echo "NODE: $SLURMD_NODENAME, CPUS: $SLURM_CPUS_PER_TASK, ARGS: $*" - -srun python "$REPO/benches/stack/bench_depth.py" "$@" --output "logs/$1_$SLURM_JOB_ID.md" diff --git a/docs/content/docs/features/gpu.mdx b/docs/content/docs/features/gpu.mdx index 847f17c..2c65813 100644 --- a/docs/content/docs/features/gpu.mdx +++ b/docs/content/docs/features/gpu.mdx @@ -15,9 +15,8 @@ uv pip install "src_method[gpu-rocm]" # ROCm On data-center GPUs behind an older driver, as on many clusters, NVIDIA's [forward-compatibility package](https://docs.nvidia.com/deploy/cuda-compatibility/) runs CUDA 13 without root: unpack `cuda-compat-13-x` and put its `compat/` -folder first on `LD_LIBRARY_PATH`. The -[Leonardo scripts](https://github.com/Algorithmiq/src-method/tree/main/benches/primitives) -do this on driver 535. +folder first on `LD_LIBRARY_PATH`. The [Leonardo](/hpc/leonardo) page does this +on driver 535. When adding `src_method` to another project's `pyproject.toml`: diff --git a/docs/content/docs/features/large-problems.mdx b/docs/content/docs/features/large-problems.mdx index 165bb01..09f175a 100644 --- a/docs/content/docs/features/large-problems.mdx +++ b/docs/content/docs/features/large-problems.mdx @@ -51,10 +51,12 @@ The sweep is planned to fit the GPU budget, counting the bytes its arrays use. CuPy's memory pool also holds blocks that are split and only partly in use, so during the sweep the pool is capped at the GPU budget plus the margin of `max(10%, 1 GiB)` of the device. That margin is the one detection leaves free, so -with a detected budget the cap is the free device memory. An estimate that falls -short by more than the margin fails at once rather than when some other -allocation does; the previous limit is restored afterwards. If a single site does not fit the -budget even with batches of one column, `src` raises `MemoryError` naming the site. +with a detected budget the cap is the free device memory; the previous limit is +restored afterwards. When the pool fragments beyond the margin, which large +batches can do, a batch that runs out of memory is computed in halves, and that +kernel keeps the finer split for the rest of the sweep; the result does not change. +If a single site does not fit the budget even with batches of one column, `src` +raises `MemoryError` naming the site. ## Inputs on disk diff --git a/docs/content/docs/hpc/index.mdx b/docs/content/docs/hpc/index.mdx new file mode 100644 index 0000000..f1bd5f8 --- /dev/null +++ b/docs/content/docs/hpc/index.mdx @@ -0,0 +1,14 @@ +--- +title: HPC clusters +description: Set up and run src_method on supercomputers, with an example job script for each. +--- + +`src_method` needs nothing cluster-specific: a Python environment with the +[GPU extra](/features/gpu), and on a node with several GPUs, +`Resources(devices=n)` (see [Large problems](/features/large-problems#several-gpus)). +What changes between clusters is the driver, the storage and the scheduler. Each +page below collects what we needed on one machine. + + + + diff --git a/docs/content/docs/hpc/leonardo.mdx b/docs/content/docs/hpc/leonardo.mdx new file mode 100644 index 0000000..520163d --- /dev/null +++ b/docs/content/docs/hpc/leonardo.mdx @@ -0,0 +1,118 @@ +--- +title: Leonardo (CINECA) +description: Run src_method on the Booster partition of Leonardo, on one GPU or all four of a node. +--- + +[Leonardo](https://docs.hpc.cineca.it/general/getting_started.html) is a EuroHPC +pre-exascale system at CINECA. Its Booster nodes have 32 Xeon 8358 cores, 512 GB +of memory and 4 A100-SXM-64GB, every pair connected by NVLink. Three things differ +from a workstation: + +- **The driver predates CUDA 13.** The nodes run driver 535, while the + `gpu-nvidia` extra installs CuPy for CUDA 13. NVIDIA's forward-compatibility + package fills the gap without root. +- **There is no local disk**, and `/tmp` is a small in-memory file system. Large + stacks, and the environments that the [out-of-core sweep](/features/large-problems) + spills, go to the Lustre scratch area, `$CINECA_SCRATCH`. +- **The home directory has a small quota.** Keep the `uv` cache and Python + installations in the project's work area. + +## Setup + +Once, on a login node, from a clone of the repository in your work area (or from +your own project that depends on `src_method[gpu-nvidia]`): + +```bash +# Keep the uv cache and Python installations out of $HOME. +export UV_CACHE_DIR=$PWD/.uv-cache UV_PYTHON_INSTALL_DIR=$PWD/.uv-python +uv sync --extra gpu-nvidia + +# Unpack NVIDIA's forward-compatible libcuda for CUDA 13. +mkdir -p .cuda-compat && cd .cuda-compat +curl -sSfO https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-compat-13-4-615.71.09-1.el8.x86_64.rpm +rpm2cpio cuda-compat-13-4-*.rpm | cpio -idm && cd .. +``` + +Add `--group bench` to `uv sync` to run the [benchmarks](/benchmarks). `saldo -b` +lists the budgets you can charge jobs to. + +## Job script + +Save this as `job.sh` in the same folder, set your account, and submit any script +with `sbatch job.sh my_script.py --its-options`. It runs from the folder you submit +from, with the four GPUs of one node visible to a single process. + +```bash +#!/usr/bin/env bash +#SBATCH --job-name=src +#SBATCH --account= +#SBATCH --partition=boost_usr_prod +#SBATCH --nodes=1 +#SBATCH --ntasks-per-node=1 +#SBATCH --cpus-per-task=32 +#SBATCH --gres=gpu:4 +#SBATCH --mem=0 +#SBATCH --exclusive +#SBATCH --time=04:00:00 +#SBATCH --output=%x_%j.out + +set -euo pipefail +cd "$SLURM_SUBMIT_DIR" + +export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK OMP_PLACES=cores OMP_PROC_BIND=spread +# No local disk, and /tmp is small: temporary files go to Lustre scratch. +export TMPDIR=$CINECA_SCRATCH/tmp +mkdir -p "$TMPDIR" + +source .venv/bin/activate +# CuPy compiles its kernels on first use; a persistent cache saves that next time. +export CUPY_CACHE_DIR=$PWD/.cupy-cache +# Driver 535 predates CUDA 13: load the forward-compatible libcuda first. +export LD_LIBRARY_PATH=$PWD/.cuda-compat/usr/local/cuda-13.4/compat${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH} + +srun python "$@" +``` + +For one GPU, use `--gres=gpu:1`. + +In the script, split the sweep among the four GPUs and spill to scratch: + +```python notest +import os + +from src_method import Resources, src + +out = src( + *stack, + chi_out=2000, + device="gpu", + resources=Resources( + devices=4, scratch_dir=os.path.join(os.environ["CINECA_SCRATCH"], "src") + ), +) +``` + +## Queues + +- **Short jobs:** for anything under 30 minutes, add + `--qos=boost_qos_dbg --time=00:30:00`. It starts quickly but runs at most two + jobs per user. +- **Longer jobs:** the default QoS can queue for hours. Priority does not depend + on the time limit, but the scheduler backfills short jobs into gaps, so a tight + `--time` starts sooner. `scontrol update JobId= TimeLimit=` can lower the + limit of a queued job, never raise it. + +## Performance notes + +- **The first run of a job is slower:** CuPy compiles its kernels (about 50 s with + an empty cache, hence `CUPY_CACHE_DIR`), and in our runs the first SRC call of a + job took a further 20 s or so, for a reason not yet isolated. +- **Lustre is shared.** Input stacks read from scratch, and environments spilled to + it, run at whatever bandwidth the file system has to spare; reading a stack once + before the sweep (e.g. `cat` its files to `/dev/null`) avoids paying for a cold + read inside the timed run. +- **Host memory sets how much reaches the disk.** The sweep keeps 90% of the + available memory as its host budget; environments beyond it go to + `scratch_dir`. The + [out-of-core benchmark](https://github.com/Algorithmiq/src-method/tree/main/benches/large) + reports where the environments of its reference runs ended up. diff --git a/docs/content/docs/hpc/meta.json b/docs/content/docs/hpc/meta.json new file mode 100644 index 0000000..e21d8ad --- /dev/null +++ b/docs/content/docs/hpc/meta.json @@ -0,0 +1,4 @@ +{ + "title": "HPC clusters", + "pages": ["leonardo"] +} diff --git a/docs/content/docs/index.mdx b/docs/content/docs/index.mdx index b70d544..1b91c25 100644 --- a/docs/content/docs/index.mdx +++ b/docs/content/docs/index.mdx @@ -29,6 +29,7 @@ the array conventions. + diff --git a/docs/content/docs/meta.json b/docs/content/docs/meta.json index a1e7709..f22ff2e 100644 --- a/docs/content/docs/meta.json +++ b/docs/content/docs/meta.json @@ -1,3 +1,3 @@ { - "pages": ["index", "getting-started", "concepts", "features", "tutorials", "api", "benchmarks", "references", "contributing"] + "pages": ["index", "getting-started", "concepts", "features", "hpc", "tutorials", "api", "benchmarks", "references", "contributing"] } diff --git a/src/src_method/_sites.py b/src/src_method/_sites.py index be2e65c..531be36 100644 --- a/src/src_method/_sites.py +++ b/src/src_method/_sites.py @@ -221,13 +221,17 @@ def _stage(self, j: int, buffer: Any) -> _Staged: # noqa: ANN401 for layer, kind in zip(self._layers, self._kinds) ] return _Staged(padded, [None] * len(padded), [], 0, None) - # A core already on the device is used as it is by a single device; a group - # stages it like any other, since it lives on one device only. + # A core already on the sweep's device is used as it is by a single device; + # a group, or a core on another device, stages it like any other. keep_device = self._group.size == 1 and not is_host(self._xp) views, device, offsets, offset = [], [], [], 0 for layer in self._layers: core = layer[j] - if keep_device and isinstance(core, self._xp.ndarray): + if ( + keep_device + and isinstance(core, self._xp.ndarray) + and core.device.id == self._group.devices[0] + ): views.append(None) device.append(core) offsets.append(offset) diff --git a/src/src_method/_sweep.py b/src/src_method/_sweep.py index 975a389..c023cd7 100644 --- a/src/src_method/_sweep.py +++ b/src/src_method/_sweep.py @@ -13,6 +13,8 @@ Every contraction runs in batches sized by `src_method._plan.make_plan` to the memory budgets. The cores are read one site at a time (`SiteSource`) and the environments live on the device, in host memory or on disk (`EnvironmentStore`). +CuPy's pool can fragment beyond the margin the plan leaves for it; a batch that +then runs out of memory is computed in smaller pieces (`_SplittingKernels`). With several devices (`DeviceGroup`), each owns a contiguous block of the sketch columns. The left-to-right pass needs no communication: the columns are @@ -26,6 +28,7 @@ import logging import threading +from math import ceil from time import perf_counter_ns from typing import TYPE_CHECKING @@ -42,12 +45,13 @@ device_pool_limit, gaussian_sketch, new_stream, + release_cached_blocks, to_numpy, truncated_qr, ) if TYPE_CHECKING: - from collections.abc import Iterator, Sequence + from collections.abc import Callable, Iterator, Sequence from types import ModuleType from numpy.typing import DTypeLike, NDArray @@ -64,6 +68,85 @@ def _ranges(n: int, batch: int) -> Iterator[tuple[int, int]]: yield lo, min(lo + batch, n) +class _SplittingKernels: + """The site kernels, computed in smaller pieces after running out of memory. + + A batch that fails is split along its batch axis and its pieces joined, so + the batches read from and written to the store stay those of the plan. The + split of a kernel lasts for the rest of the sweep. One instance per device. + + Args: + kernels: The kernels of the sweep. + xp: Array module (``numpy`` or ``cupy``). + tag: Prefix of the log messages. + """ + + def __init__(self, kernels: SiteKernels, xp: ModuleType, tag: str) -> None: + self._kernels, self._xp, self._tag = kernels, xp, tag + self._pieces: dict[str, int] = {} + + def env(self, env: NDArray, omega: NDArray, cores: Sequence[NDArray]) -> NDArray: + """`SiteKernels.env`, split along the sketch columns.""" + return self._split( + "env", + env.shape[0], + lambda a, b: self._kernels.env(env[a:b], omega[a:b], cores), + axis=0, + ) + + def sketch(self, env: NDArray, cores: Sequence[NDArray], proj: NDArray) -> NDArray: + """`SiteKernels.sketch`, split along the sketch columns.""" + return self._split( + "sketch", + env.shape[0], + lambda a, b: self._kernels.sketch(env[a:b], cores, proj), + axis=-1, + ) + + def project(self, eta: NDArray, cores: Sequence[NDArray], proj: NDArray) -> NDArray: + """`SiteKernels.project`, split along the rows.""" + return self._split( + "project", + eta.shape[0], + lambda a, b: self._kernels.project(eta[a:b], cores, proj), + axis=0, + ) + + def first(self, cores: Sequence[NDArray], proj: NDArray) -> NDArray: + """`SiteKernels.first`, split along the rows.""" + return self._split( + "first", + proj.shape[0], + lambda a, b: self._kernels.first(cores, proj[a:b]), + axis=-3, + ) + + def _split( + self, kernel: str, n: int, compute: Callable[[int, int], NDArray], *, axis: int + ) -> NDArray: + parts, start = [], 0 + while start < n: + pieces = self._pieces.get(kernel, 1) + stop = min(n, start + ceil(n / pieces)) + try: + parts.append(compute(start, stop)) + except MemoryError: + if stop - start == 1: + raise + release_cached_blocks(self._xp) + self._pieces[kernel] = 2 * pieces + logger.debug( + "%sOut of memory in a %s batch of %d rows; %d pieces from now on", + self._tag, + kernel, + stop - start, + 2 * pieces, + ) + continue + start = stop + return parts[0] if len(parts) == 1 else self._xp.concatenate(parts, axis=axis) + + def sweep( layers: Sequence[Sequence[Site]], kind: TrainKind, @@ -203,7 +286,7 @@ def run(self, rank: int, source: SiteSource) -> list[NDArray] | None: xp, tag = self.group.xp, self._tag(rank) lo, hi = self.columns[rank] env_shapes = [(hi - lo, *(s[0] for s in site)) for site in self.shapes] - kernels = SiteKernels(self.depth) + kernels = _SplittingKernels(SiteKernels(self.depth), xp, tag) with ( device_pool_limit(xp, self.budgets.device_cap), EnvironmentStore( @@ -237,7 +320,7 @@ def run(self, rank: int, source: SiteSource) -> list[NDArray] | None: def _left_to_right( self, rank: int, - kernels: SiteKernels, + kernels: _SplittingKernels, source: SiteSource, store: EnvironmentStore, ) -> None: @@ -260,7 +343,7 @@ def _left_to_right( def _right_to_left( self, rank: int, - kernels: SiteKernels, + kernels: _SplittingKernels, source: SiteSource, store: EnvironmentStore, ) -> list[NDArray] | None: diff --git a/src/src_method/utils/__init__.py b/src/src_method/utils/__init__.py index c631795..54039b8 100644 --- a/src/src_method/utils/__init__.py +++ b/src/src_method/utils/__init__.py @@ -20,6 +20,7 @@ is_host, new_stream, pinned_empty, + release_cached_blocks, sketch_dtype, to_device_async, to_host_async, @@ -46,6 +47,7 @@ "is_host", "new_stream", "pinned_empty", + "release_cached_blocks", "sketch_dtype", "to_device_async", "to_host_async", diff --git a/src/src_method/utils/_backend.py b/src/src_method/utils/_backend.py index 9dd6129..9a98f79 100644 --- a/src/src_method/utils/_backend.py +++ b/src/src_method/utils/_backend.py @@ -318,6 +318,16 @@ def device_pool_bytes(xp: ModuleType) -> int: return xp.get_default_memory_pool().total_bytes() +def release_cached_blocks(xp: ModuleType) -> None: + """Return the free blocks cupy's pool caches on the current device to CUDA. + + Blocks split for smaller allocations stay cached until all their parts are + free. No-op on the host. + """ + if not is_host(xp): + xp.get_default_memory_pool().free_all_blocks() + + @contextmanager def device_pool_limit(xp: ModuleType, budget: int | None) -> Iterator[None]: """Cap cupy's default memory pool at ``budget`` bytes beyond its current use. diff --git a/tests/test_gpu_backend.py b/tests/test_gpu_backend.py index a33652b..6e7c031 100644 --- a/tests/test_gpu_backend.py +++ b/tests/test_gpu_backend.py @@ -397,6 +397,26 @@ def test_two_gpus_take_device_inputs() -> None: ) +@pytest.mark.usefixtures("two_gpus") +@pytest.mark.usefixtures("two_gpus") +def test_one_gpu_takes_inputs_from_another() -> None: + """Cores given on GPU 1 are staged to GPU 0 when the sweep runs there alone.""" + rng = np.random.default_rng(11) + stack = [random_mpo_arrays([2, 3, 3, 2], rng) for _ in range(2)] + with cupy.cuda.Device(1): + on_device = [[cupy.asarray(t) for t in train] for train in stack] + + host = src(*stack, chi_out=8, seed=0, device="gpu") + other = src( + *on_device, chi_out=8, seed=0, device="gpu", resources=Resources(devices=[0]) + ) + + reference = dense_mpo(host) + assert ( + np.linalg.norm(dense_mpo(other) - reference) / np.linalg.norm(reference) < 1e-10 + ) + + @pytest.mark.usefixtures("two_gpus") def test_two_gpus_raise_the_first_error(monkeypatch: pytest.MonkeyPatch) -> None: """An error on one GPU stops both, restores the pool limits and is raised.""" diff --git a/tests/test_stack.py b/tests/test_stack.py index b7d0c38..404caa5 100644 --- a/tests/test_stack.py +++ b/tests/test_stack.py @@ -1,5 +1,6 @@ """Test `src` over stacks of tensor trains, and the sweep behind it.""" +import logging import threading from functools import reduce @@ -409,6 +410,52 @@ def test_compress_passes_resources(rng): compress(A, chi_out=8, resources=Resources(host_memory="1kB")) +def _rows_capped(monkeypatch, cap): + """Make every kernel run out of memory on batches of more than ``cap`` rows.""" + rows_of = { + "env": lambda args: args[0].shape[0], + "sketch": lambda args: args[0].shape[0], + "project": lambda args: args[0].shape[0], + "first": lambda args: args[1].shape[0], + } + for name, rows in rows_of.items(): + original = getattr(SiteKernels, name) + + def kernel(self, *args, original=original, rows=rows): + if rows(args) > cap: + msg = "simulated out of memory" + raise MemoryError(msg) + return original(self, *args) + + monkeypatch.setattr(SiteKernels, name, kernel) + + +@pytest.mark.parametrize("devices", [None, 2]) +def test_out_of_memory_splits_the_batches(devices, rng, monkeypatch, caplog): + stack = [random_mpo([3, 4, 4, 4, 3], rng) for _ in range(3)] + resources = Resources(devices=devices) + reference = src(*stack, chi_out=16, seed=4, dtype=np.complex128) + + _rows_capped(monkeypatch, 4) + with caplog.at_level(logging.DEBUG, logger="src_method"): + split = src( + *stack, chi_out=16, seed=4, dtype=np.complex128, resources=resources + ) + + assert [t.shape for t in split] == [t.shape for t in reference] + assert rel_error(split, dense(reference)) < 1e-10 + for kernel in ("env", "sketch", "project", "first"): + assert f"Out of memory in a {kernel} batch" in caplog.text + + +def test_out_of_memory_on_one_row_raises(rng, monkeypatch): + stack = [random_mpo([3, 4, 4, 4, 3], rng) for _ in range(3)] + + _rows_capped(monkeypatch, 0) + with pytest.raises(MemoryError, match="simulated out of memory"): + src(*stack, chi_out=16, seed=4) + + # ------------------------------- # --- Several (simulated) devices --- # -------------------------------