VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systems

---

Overview

We propose VerifyMAS, a hypothesis verification framework for agent failure attribution. Instead of directly predicting faulty agents and error types, VerifyMAS formulates and verifies failure hypotheses against full trajectories. This verification-based approach decomposes attribution into trajectory-level error validation and fine-grained agent localization, providing an error-first attribution approach that captures global failure patterns while substantially reducing the search space. We further introduce a hypothesis-based data construction strategy grounded in a structured error taxonomy and fine-tune a specialized LLM verifier model for trajectory-level failure verification and agent attribution. Experiments on Aegis-Bench and Who&When show that VerifyMAS consistently improves diverse backbone models, including open-source Qwen and API-based GPT models, outperforming prior methods without sacrificing inference efficiency for long multi-agent trajectories.

Main Results

Repository Structure

The main branch is organized around data, SFT data preparation, baseline inference, and zero-shot evaluation. VerifyMAS verifies failure hypotheses against full multi-agent trajectories to identify errors and responsible agents.

Directory or file	Contents and purpose
baselines/	Baseline inference scripts, including CoT and error-first variants, plus prompt.txt and prompt_cot.txt
data/	Two JSONL files of test datasets: test_aegis.jsonl and whowhen.jsonl
sft_data_construction/	Scripts for converting multi-agent JSONL data, including one that handles rare agents
training_data/	The training datasets and log for SFT
zero_shot/	Inference and evaluation scripts for standard evaluation, SFT-model evaluation, and OpenAI API inference

The processed training data for VerifyMAS can be downloaded from this Google Drive folder or Hugging face. The model can be downloaed from Hugging face.

📖 Citation

If you find this work useful, please cite our paper:

@article{qiao2026verifymas,
  title={VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systems},
  author={Qiao, Hezhe and Tong, Hanghang and Lim, Ee-Peng and Liu, Bing and Pang, Guansong},
  journal={arXiv preprint arXiv:2605.17467},
  year={2026}
}