# Benchmark data

We do not redistribute any third-party benchmark. Run

```bash
bash scripts/download_data.sh
```

from the repository root to fetch everything that has a direct official URL.
Expected layout after download:

```
data/
  bfcl/        BFCL_v4_simple_python.json  BFCL_v4_multiple.json  BFCL_v4_parallel.json
  sealtools/   tool.jsonl  test_in_domain.jsonl
  toolalpaca/  eval_simulated.json  eval_real.json
  apibank/     level-1-api.json  level-2-api.json
  toolbench/   test_instruction/G1_instruction.json ... (manual, see below)
  tau2/        tools_airline.json  tools_retail.json  tools_telecom.json (generated, see below)
```

## Provenance and licenses

| Dataset | Source | License |
|---|---|---|
| BFCL v4 | [ShishirPatil/gorilla](https://github.com/ShishirPatil/gorilla) (`berkeley-function-call-leaderboard/bfcl_eval/data`) | Apache-2.0 |
| Seal-Tools | [fairyshine/Seal-Tools](https://github.com/fairyshine/Seal-Tools) (`Seal-Tools_Dataset`) | Apache-2.0 |
| ToolAlpaca | [tangqiaoyu/ToolAlpaca](https://github.com/tangqiaoyu/ToolAlpaca) (`data/`) | Apache-2.0 |
| API-Bank | [HF: liminghao1630/API-Bank](https://huggingface.co/datasets/liminghao1630/API-Bank) (`test-data/`) | MIT |
| ToolBench | [OpenBMB/ToolBench](https://github.com/OpenBMB/ToolBench) official Google Drive release | Apache-2.0 |
| tau2-bench | [sierra-research/tau2-bench](https://github.com/sierra-research/tau2-bench) | MIT |

## Manual steps

**ToolBench** (only needed for the 4-benchmark main table,
`harness/phase4_maintable.py`): download the data archive linked from the
[OpenBMB/ToolBench README](https://github.com/OpenBMB/ToolBench) (Google
Drive) and copy `data/test_instruction/G{1,2,3}_*.json` to
`data/toolbench/test_instruction/`.

**tau2-bench** (only needed for the tau2 experiments): the harness needs the
per-domain tool registries `data/tau2/tools_{domain}.json`. These are
generated from the tau2-bench repo's domain toolkit sources
(`src/tau2/domains/<domain>/tools.py`) with `harness/tau2_extract.py`, which
parses the `@is_tool` methods into BFCL-style JSON schemas. Live trace
generation additionally requires installing
[tau2-bench](https://github.com/sierra-research/tau2-bench) itself (the
`tau2` CLI) — see `harness/tau2_live.py`.