BioProBench

---

📢 Latest News

• 🛡️ [2026-07] Data Compliance & Robustness Upgrade! We have rigorously refined our corpus to ensure 100% open-source compliance, resulting in a high-fidelity foundational collection of 22,413 human-written protocols.
• ✨ [2026-03] Data Split Update! We have officially released the Train/Test splits for each task (PQA, ORD, ERR, GEN, REA), making it easier for the community to train and evaluate models consistently.
• 🔥 [2026-03] Our BioProAgent is now live on AI4S LAB! Try it out and order wet-lab experiments here.
• 🎉 [2026-03] Our BioProAgent has been accepted by the ICML 2026 LLA Workshop!
• 📝 [2026-01] BioProBench paper has been updated with new experimental results on frontier models (GPT-5.4, Gemini 3.1 Pro, Claude 4.6 Opus). Read the latest on arXiv.

---

🌟 Introduction

The realization of autonomous scientific experimentation is currently limited by LLMs' struggle to grasp the strict procedural logic and accuracy required by biological protocols.

Enter BioProBench—the first large-scale, integrated multi-task benchmark dedicated specifically to procedural reasoning in biological contexts. Moving beyond basic declarative knowledge and simple QA, BioProBench is designed to test the structured, causal, and conditional logic required for true scientific automation.

🚀 Key Features

• 📚 Unprecedented Scale: Grounded in 22,413 professionally authored, human-written protocols.
• 🧩 Rich Task Diversity: Systematically constructed into a dataset of 523,784 structured task instances.
• 🎯 5 Core Procedural Tasks: Covers Protocol Question Answering (PQA), Step Ordering (ORD), Error Correction (ERR), Protocol Generation (GEN), and Protocol Reasoning (REA).
• 🧬 Broad Domain Coverage: Spans 16 biological subfields sourced from 5 authoritative repositories.
• 🔬 Standardized Evaluation: Features a rigorous benchmark utilizing novel, domain-specific metrics (e.g., Step Recall, Step Precision) to accurately quantify structural fidelity.

---

📊 Dataset Structure & Tasks

We provide standardized JSON files for each task, meticulously partitioned into Train and Test splits to facilitate model fine-tuning and rigorous evaluation:

Note: Data derived from openly licensed sources is publicly available. Please refer to our licensing details for usage terms.

---

🔬 Key Findings from Frontier LLMs

We evaluated 10 mainstream state-of-the-art LLMs, alongside a suite of the latest 2026 frontier models. Our rigorous evaluation reveals:

• The Comprehension Plateau: While top-tier models excel at basic declarative comprehension, performance degrades significantly on tasks demanding deep reasoning, quantitative precision, and safety awareness.
• The Generation Bottleneck: Models struggle profoundly with long-form procedural generation (GEN task). Even the most advanced models omit necessary experimental steps, underscoring that open-ended procedural planning remains a formidable challenge.
• The Power of Grounding: To mitigate these issues, we introduce ProAgent. Grounded in our BioProCorpus via a Retrieval-Augmented Generation (RAG) framework, ProAgent substantially advances the state-of-the-art, significantly improving procedural step recall and reasoning accuracy.

---

🔗 Useful Links & Resources

• 🌐 Project Website: Explore BioProBench
• 💻 GitHub Repository: bioprotocolbench (Evaluation scripts, training code, and ProAgent implementation)

---

🤝 Contributing & Contact

We welcome contributions from the community to help pave the way for reliable AI in autonomous science! Whether it's suggesting new protocol sources, expanding biological domains, or proposing novel procedural tasks, your input is valued.

• Email: sunshineliuyuyang@gmail.com
• Issues: Feel free to open an issue or submit a PR on our GitHub.

📜 Citation

If you find BioProBench or ProAgent useful in your research, please consider citing our work:

@inproceedings{liu2026bioprobench,
  title={BioProBench: A Corpus and Benchmark for Biological Protocol Reasoning in Autonomous Science},
  author={Liu, Yuyang and Lv, Liuzhenghao and Zhang, Xiancheng and Wang, Jingya and Yuan, Li and Tian, Yonghong},
  booktitle={Proceedings of the 43rd International Conference on Machine Learning (ICML)},
  year={2026}
}