| # Benchmark data |
|
|
| We do not redistribute any third-party benchmark. Run |
|
|
| ```bash |
| bash scripts/download_data.sh |
| ``` |
|
|
| from the repository root to fetch everything that has a direct official URL. |
| Expected layout after download: |
|
|
| ``` |
| data/ |
| bfcl/ BFCL_v4_simple_python.json BFCL_v4_multiple.json BFCL_v4_parallel.json |
| sealtools/ tool.jsonl test_in_domain.jsonl |
| toolalpaca/ eval_simulated.json eval_real.json |
| apibank/ level-1-api.json level-2-api.json |
| toolbench/ test_instruction/G1_instruction.json ... (manual, see below) |
| tau2/ tools_airline.json tools_retail.json tools_telecom.json (generated, see below) |
| ``` |
|
|
| ## Provenance and licenses |
|
|
| | Dataset | Source | License | |
| |---|---|---| |
| | BFCL v4 | [ShishirPatil/gorilla](https://github.com/ShishirPatil/gorilla) (`berkeley-function-call-leaderboard/bfcl_eval/data`) | Apache-2.0 | |
| | Seal-Tools | [fairyshine/Seal-Tools](https://github.com/fairyshine/Seal-Tools) (`Seal-Tools_Dataset`) | Apache-2.0 | |
| | ToolAlpaca | [tangqiaoyu/ToolAlpaca](https://github.com/tangqiaoyu/ToolAlpaca) (`data/`) | Apache-2.0 | |
| | API-Bank | [HF: liminghao1630/API-Bank](https://huggingface.co/datasets/liminghao1630/API-Bank) (`test-data/`) | MIT | |
| | ToolBench | [OpenBMB/ToolBench](https://github.com/OpenBMB/ToolBench) official Google Drive release | Apache-2.0 | |
| | tau2-bench | [sierra-research/tau2-bench](https://github.com/sierra-research/tau2-bench) | MIT | |
|
|
| ## Manual steps |
|
|
| **ToolBench** (only needed for the 4-benchmark main table, |
| `harness/phase4_maintable.py`): download the data archive linked from the |
| [OpenBMB/ToolBench README](https://github.com/OpenBMB/ToolBench) (Google |
| Drive) and copy `data/test_instruction/G{1,2,3}_*.json` to |
| `data/toolbench/test_instruction/`. |
|
|
| **tau2-bench** (only needed for the tau2 experiments): the harness needs the |
| per-domain tool registries `data/tau2/tools_{domain}.json`. These are |
| generated from the tau2-bench repo's domain toolkit sources |
| (`src/tau2/domains/<domain>/tools.py`) with `harness/tau2_extract.py`, which |
| parses the `@is_tool` methods into BFCL-style JSON schemas. Live trace |
| generation additionally requires installing |
| [tau2-bench](https://github.com/sierra-research/tau2-bench) itself (the |
| `tau2` CLI) — see `harness/tau2_live.py`. |
|
|
