data/README.md

Benchmark data

We do not redistribute any third-party benchmark. Run

bash scripts/download_data.sh

from the repository root to fetch everything that has a direct official URL. Expected layout after download:

data/
  bfcl/        BFCL_v4_simple_python.json  BFCL_v4_multiple.json  BFCL_v4_parallel.json
  sealtools/   tool.jsonl  test_in_domain.jsonl
  toolalpaca/  eval_simulated.json  eval_real.json
  apibank/     level-1-api.json  level-2-api.json
  toolbench/   test_instruction/G1_instruction.json ... (manual, see below)
  tau2/        tools_airline.json  tools_retail.json  tools_telecom.json (generated, see below)

Provenance and licenses

Dataset	Source	License
BFCL v4	ShishirPatil/gorilla (berkeley-function-call-leaderboard/bfcl_eval/data)	Apache-2.0
Seal-Tools	fairyshine/Seal-Tools (Seal-Tools_Dataset)	Apache-2.0
ToolAlpaca	tangqiaoyu/ToolAlpaca (data/)	Apache-2.0
API-Bank	HF: liminghao1630/API-Bank (test-data/)	MIT
ToolBench	OpenBMB/ToolBench official Google Drive release	Apache-2.0
tau2-bench	sierra-research/tau2-bench	MIT

Manual steps

ToolBench (only needed for the 4-benchmark main table, harness/phase4_maintable.py): download the data archive linked from the OpenBMB/ToolBench README (Google Drive) and copy data/test_instruction/G{1,2,3}_*.json to data/toolbench/test_instruction/.

tau2-bench (only needed for the tau2 experiments): the harness needs the per-domain tool registries data/tau2/tools_{domain}.json. These are generated from the tau2-bench repo's domain toolkit sources (src/tau2/domains/<domain>/tools.py) with harness/tau2_extract.py, which parses the @is_tool methods into BFCL-style JSON schemas. Live trace generation additionally requires installing tau2-bench itself (the tau2 CLI) — see harness/tau2_live.py.