A full-catalog benchmark and evaluation suite for bundle completion: observe part of a bundle (a playlist, an outfit, a meal, a shopping basket) and predict the items that complete it, scored against the whole catalog with no negative sampling.
BundleBench exists because the standard bundle-completion protocol scores each held-out item against a small shortlist of sampled negatives, a pool that already contains the ground truth and whose uniform negatives are popularity-trivial. Scored full-catalog instead, the reported standing of generative bundle construction reverses: the state-of-the-art discrete-diffusion model (DDBC) collapses from a reproduced set-level F1 of 0.30 to 0.033, trivial item-item co-occurrence beats it on both of its own benchmarks, and both of its learned baseline families (SASRec, a one-shot set completer) collapse on every adequately powered catalog. This package is the runnable artifact behind that finding, and the honest target for whatever comes next: beat the simple feature-free family full-catalog and pass the diagnostics while doing it.
What ships here: the full-catalog protocol with fixed deterministic splits over 11 bundle catalogs, six reference baselines plus the generative (DDBC decode) and learned (SASRec, SetCompleter) families, six metrics, four reusable protocol diagnostics, a pre-registered significance harness, per-seed result files behind every table in the paper, and a leaderboard that accepts entries by pull request.
pip install bundlebench # feature-free core (numpy / scipy / scikit-learn)
pip install "bundlebench[neural]" # + torch, for the learned baselines and DDBC adapter
pip install "bundlebench[content]" # + sentence-transformers, for the content-reranker armFrom source:
git clone https://github.com/anisiolacerda/bundlebench && cd bundlebench
pip install -e ".[neural,dev]"# feature-free leaderboard on one catalog (see DATASHEET.md for every download link)
python -m bundlebench --dataset youshu --path PATH/TO/bundle_item.txt --format edgelist
# add the learned families (SASRec + ID-only SetCompleter); GPU recommended
python -m bundlebench --dataset youshu --path PATH/TO/bundle_item.txt --format edgelist \
--neural --device cuda --out results/_working/youshu.jsonProgrammatic use:
from bundlebench.leaderboard import run_leaderboard
res = run_leaderboard("youshu", "PATH/bundle_item.txt", "edgelist", neural=True)
print(res["means"]["cooccurrence"]["f1"])
print(res["vs_cooc"]["cooc_gt_sasrec"]["f1"]["p_value"]) # paired Wilcoxon, per test bundleA bundle is a set of catalog items. For each held-out bundle the harness observes half the items
(ceil(|B|/2)) and predicts the rest, ranking every catalog item with no sampled negatives.
Breadth catalogs use bundles at |B| >= 4 so both halves carry at least 2 items (the threshold is
not load-bearing; a --min-size 2 re-run reproduces every conclusion). Splits are deterministic
in --seed. Metrics are DDBC's own set-level family (F1, Jaccard, and OAS where item features
exist) plus full-catalog ranking metrics (hit@1, MRR, recall@k), reported per test bundle so
any two methods compare with a per-example paired Wilcoxon test.
bundlebench.protocol.evaluate is Algorithm 1 in the paper. It takes any scorer unchanged: an
object with a name and a score(observed) -> np.ndarray method returning a full-catalog score
vector f(S) in R^N. Only the candidate pool differs from the standard shortlist protocol, so any
change in standing is attributable to the protocol alone.
| family | method | module |
|---|---|---|
| feature-free | popularity, co-occurrence, item-kNN, PPMI-SVD CF | bundlebench.baselines |
| feature-free scaffold | retrieve-then-rerank (+ random / content ablation arms) | bundlebench.rerank |
| generative SOTA | DDBC decode, full-catalog re-scoring | bundlebench.ddbc |
| learned, sequential | SASRec (one-shot and sequential-greedy construction) | bundlebench.models.sasrec |
| learned, non-sequential | SetCompleter (ID-only where no content features exist) | bundlebench.models.setcompleter |
Every method conditions on the identical observed seed and is scored over the identical full catalog; all training uses training bundles only.
One-call checks a submitter runs to show a result is real, not a protocol artifact:
- Shortlist-to-full-catalog inflation. A method's hit@1 under its own rho-pool shortlist vs full-catalog; the ratio quantifies the over-claim (up to ~17x in the paper).
- Popularity-triviality. How well popularity alone does inside a random-negative shortlist. If it scores well, the protocol is rewarding popularity, not completion.
- Sampler-noise null (
decode_tau0). For stochastic decoders: the fraction of completed items that change between two decode runs, compared against the change from ablating the term under test. Separates "in the architecture" from "exercised at inference"; this check caught a false positive in our own development that a static independence test had passed. - Oracle re-rank ceiling. The best full-catalog recall an ideal re-ranking of a method's own top-M candidates could reach. Separates retrieval failure from ranking failure and bounds any reranker.
BundleBench is a benchmarking tool and evaluation methodology, not a new dataset. It contributes
the fixed observed/held-out splits, the harness, and the diagnostics over existing public bundle
catalogs, and redistributes no third-party data. DATASHEET.md documents every catalog (source,
license, access link verified at release, fields consumed, split construction). Summary:
| dataset | domain | items | test | loader |
|---|---|---|---|---|
| Spotify-90 | playlist | 254,155 | 969 | paper pipeline (gated MPD; see datasheet) |
| POG_k=4 | outfit | 31,217 | 969 | paper pipeline (public download) |
| MealRec+H / +L | meal | 7,280 / 10,589 | 4,677 / 1,181 | edgelist |
| Youshu | book | 28,074 | 477 | edgelist |
| NetEase | playlist | 123,628 | 2,000 | edgelist |
| iFashion | outfit | 42,563 | 1,741 | edgelist |
| Steam | game | 2,787 | 28 (underpowered) | steam |
| BundleRec clothing / electronic / food | consumer | 4,487 / 3,499 / 3,767 | 58 / 60 / 67 | csv |
The nine feature-free catalogs need no gated data and no item features. The core findings reproduce without any gated source: the co-occurrence-over-DDBC overturn end-to-end on POG, and the breadth overturn on seven public catalogs.
| paper table / claim | command |
|---|---|
| Breadth leaderboard (Table 3) | DATA=... bash scripts/run_breadth.sh --neural --device cuda |
| Home-benchmark leaderboard, DDBC rows (Table 2) | python scripts/eval_ddbc_fullcatalog.py ... with the checkpoints + token caches from the paper's DOI archive; decode config in configs/ddbc_decode.yaml |
| Home-benchmark SASRec / SetCompleter rows (Table 2) | python -m bundlebench on the DDBC splits; exact run configs in results/README.md |
| Retrieve-then-rerank on MealRec (Table 4, Finding 5) | python scripts/run_rerank_mealrec.py --path ... --seeds 5 |
| rho-inflation sweep (Appendix C.1) | python scripts/ablation_rho_sweep.py ... |
| Reranker component ablation (Appendix C.2) | run_rerank_mealrec.py with --emb-source {cf,random,content} --svd-dim {32,64,128,256} --top-m {20,50,100,200} --retrieval {cooc,pop} |
| DDBC decode-budget sweep (Appendix C.3) | eval_ddbc_fullcatalog.py over the archived per-step decode outputs |
| SASRec construction / threshold ablation (Appendix C.4) | python scripts/ablation_sasrec_construction.py ... |
The per-seed result files behind every table are committed under results/ (see
results/README.md for the file-to-claim map). The trained DDBC checkpoints and RVQ token caches
exceed repository hosting and are archived with a DOI alongside the paper (ACM "Artifacts
Available" badge target).
A submission implements the scorer interface (score(observed) -> R^N) and follows the workflow
in CONTRIBUTING.md:
- Tune only on the harness's validation partition:
--val-frac 0.1 --split val(a deterministic holdout carved from train, disjoint from test). Tuning on the fixed test bundles is exactly the practice this partition exists to prevent. - Run Protocol B once on the fixed test split.
- Report the four diagnostics alongside the leaderboard metrics, with per-seed result files.
- Open a pull request adding your entry to
LEADERBOARD.mdand your result files underresults/.
Per-example paired Wilcoxon over test bundles (ties dropped, effective n reported; for binary
hit@1 this reduces to an exact sign test on discordant pairs), one-sided in the pre-registered
direction, Bonferroni-corrected across the paper's m=29 primary comparisons. A percentile
bootstrap-CI utility ships alongside. Every p-value in the paper reproduces from the per-seed
files in results/ with this module.
src/bundlebench/ the pip-installable package (protocol, metrics, baselines, rerank,
ddbc adapter, neural families, diagnostics, stats, leaderboard, CLI)
tests/ unit tests (pytest)
scripts/ table-level reproduction scripts
configs/ the exact DDBC decode configuration
results/ committed per-seed result files behind every paper table
DATASHEET.md per-catalog provenance, licenses, verified access links
LEADERBOARD.md the corrected full-catalog leaderboard; entries by PR
CONTRIBUTING.md the submission workflow
The benchmark consumes only bundle-item memberships and item metadata, which carry no user
identifiers. Several source releases additionally contain pseudonymous user-interaction records;
BundleBench neither uses nor redistributes those fields (per-catalog details in DATASHEET.md).
No copyrighted media is redistributed. The code is MIT-licensed; each dataset remains governed by
its source license (research-only where so licensed), and use of the benchmark is restricted to
non-commercial research consistent with those licenses.
@inproceedings{bundlebench,
title = {BundleBench: A Full-Catalog Benchmark and Evaluation Suite for Bundle Completion},
booktitle = {Submitted to the 33rd ACM SIGKDD Conference on Knowledge Discovery and Data
Mining (Datasets and Benchmarks Track)},
year = {2027}
}