LabAgent

This is the official code repo for manuscript: LabAgent: Customize any Research Hubs for Scientific Discoveries Using AI Agents

LabAgent reads a repository or a paper, writes a skill from what it finds, and then reproduces the work that skill describes. The two halves are separate programs and can be used separately: labagent.explore produces skills, labagent.reproduce consumes them.

Installation

Python 3.10 or newer.

conda env create -f labagent.yml      # optional, pins the versions used here
conda activate labagent

pip install -e .

The key comes from the environment; nothing here reads it from a file.

export ANTHROPIC_API_KEY=...

Optional extras:

pip install -e ".[literature]"   # Google Scholar search in the explore stage

The reproduce stage runs on mini-swe-agent and litellm; both install with the package. When a run starts, the skill directory is copied to .skill/ inside the run's working directory, which is where the agent is told to read its loci from.

TDC installs separately. On Python 3.13 pip install PyTDC builds an old scikit-learn from source and fails.

pip install --no-deps PyTDC

Skills

A skill is what this system produces and what it consumes. skills/ holds the sixty-three used in the manuscript, one directory each, keyed by name:

directory	skills	over
skills/tdc/	11	TDC ADMET leaderboard entries
skills/biomnibench/	16	one per BiomniBench task type
skills/genebench/	10	one per GeneBench-Pro problem
skills/proteingym/	26	one per ProteinGym predictor

Each carries a SKILL.md with its frontmatter, an index, and a loci/ directory of the sections the agent reads. The -comprehensive suffix is not decoration: a skill is generated broad and narrowed later, and the name records which stage it is at.

skills/flatten_skills.py reshapes a freshly generated tree into the flat layout the loader expects: one directory per skill, SKILL.md at its root. Skills generated by labagent.explore come out nested and need it.

Writing a skill

python -m labagent.explore website --repo owner/repo
python -m labagent.explore website --urls https://example.com

Reproducing with a skill

python -m labagent.reproduce run --skill caliciboost \
    --skills-dir skills/tdc --dataset /path/to/data
python -m labagent.reproduce list --skills-dir skills/tdc

--skill names the skill; without it, auto asks the model to choose from what --skills-dir holds.

Datasets

Please check benchmarks/README.md for downloading and placing the four benchmarks: TDC ADMET, BiomniBench-DA, GeneBench-Pro and ProteinGym.

To use UK Biobank genotypes for the SuSiE2 case study, please visit this website to request access to protected data.

Evaluation

evaluation/ holds the scorer for each benchmark, so that a reported score can be checked rather than taken on trust: how partial credit is awarded, and what each claim in a rubric is worth.

file	benchmark
score_biomnibench.py	BiomniBench-DA
score_genebench.py	GeneBench-Pro
score_proteingym.py	ProteinGym
score_susie2.py	SuSiE2 case study

BiomniBench is scored by keyword coverage, not by the benchmark's own LLM judge. The output field is called exact_match_score, which is not what it computes: it measures keyword overlap between the rubric's A-level description and the agent's trace and answer. The field keeps its old name so that earlier plotting code still reads it. A judge that calls a third-party model gives a number nobody can reproduce without that model and that key; keyword coverage is deterministic and recomputable from the artifacts in a run. The manuscript reports it as a keyword-coverage score and discusses its limitations.

The scorer has one mode, the one the manuscript reports. Normalisation, the plural fallback, the penalty-criterion default and the answer-only diagnostic were all options once; the runs behind the paper had them off, so they are gone rather than left as flags whose only correct setting is the one we used. Verified against the original run: fifty tasks, zero differing scores.

python evaluation/score_biomnibench.py \
    --official-root <BiomniBench-DA> \
    --runs-root <your runs> \
    --out-json scores.json

Keywords are matched on the trace and the answer together, level A at a match ratio of 0.5 and level B at 0.25.

score_genebench.py collects one verifier/reward.json per run; those files are written by GeneBench-Pro's own reference_grader.py, which comes with that package rather than with this repository.

score_susie2.py also scans a completed run for signs that it read the original authors' precomputed output instead of running the simulation. Give it --author-id with the account name whose tree must not be read from; the result is reported, never used to withhold a score.

Acknowledgements

The benchmarks this work is measured on are other people's: the Therapeutics Data Commons, ProteinGym, BiomniBench-DA, and GeneBench-Pro. The agent's execution loop builds on mini-swe-agent. Thanks to all of them for their code and their data.