File size: 2,265 Bytes
a484e22 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | # Benchmark data
We do not redistribute any third-party benchmark. Run
```bash
bash scripts/download_data.sh
```
from the repository root to fetch everything that has a direct official URL.
Expected layout after download:
```
data/
bfcl/ BFCL_v4_simple_python.json BFCL_v4_multiple.json BFCL_v4_parallel.json
sealtools/ tool.jsonl test_in_domain.jsonl
toolalpaca/ eval_simulated.json eval_real.json
apibank/ level-1-api.json level-2-api.json
toolbench/ test_instruction/G1_instruction.json ... (manual, see below)
tau2/ tools_airline.json tools_retail.json tools_telecom.json (generated, see below)
```
## Provenance and licenses
| Dataset | Source | License |
|---|---|---|
| BFCL v4 | [ShishirPatil/gorilla](https://github.com/ShishirPatil/gorilla) (`berkeley-function-call-leaderboard/bfcl_eval/data`) | Apache-2.0 |
| Seal-Tools | [fairyshine/Seal-Tools](https://github.com/fairyshine/Seal-Tools) (`Seal-Tools_Dataset`) | Apache-2.0 |
| ToolAlpaca | [tangqiaoyu/ToolAlpaca](https://github.com/tangqiaoyu/ToolAlpaca) (`data/`) | Apache-2.0 |
| API-Bank | [HF: liminghao1630/API-Bank](https://huggingface.co/datasets/liminghao1630/API-Bank) (`test-data/`) | MIT |
| ToolBench | [OpenBMB/ToolBench](https://github.com/OpenBMB/ToolBench) official Google Drive release | Apache-2.0 |
| tau2-bench | [sierra-research/tau2-bench](https://github.com/sierra-research/tau2-bench) | MIT |
## Manual steps
**ToolBench** (only needed for the 4-benchmark main table,
`harness/phase4_maintable.py`): download the data archive linked from the
[OpenBMB/ToolBench README](https://github.com/OpenBMB/ToolBench) (Google
Drive) and copy `data/test_instruction/G{1,2,3}_*.json` to
`data/toolbench/test_instruction/`.
**tau2-bench** (only needed for the tau2 experiments): the harness needs the
per-domain tool registries `data/tau2/tools_{domain}.json`. These are
generated from the tau2-bench repo's domain toolkit sources
(`src/tau2/domains/<domain>/tools.py`) with `harness/tau2_extract.py`, which
parses the `@is_tool` methods into BFCL-style JSON schemas. Live trace
generation additionally requires installing
[tau2-bench](https://github.com/sierra-research/tau2-bench) itself (the
`tau2` CLI) — see `harness/tau2_live.py`.
|