xingxm commited on
Commit
4fda809
·
verified ·
1 Parent(s): e98071d

Document per-case score and verdict files

Browse files
Files changed (1) hide show
  1. README.md +17 -1
README.md CHANGED
@@ -112,12 +112,28 @@ Two observations that only the full rubric set exposes:
112
 
113
  | File | Content |
114
  |---|---|
 
 
115
  | [`eval/benchmark_summary_v592.csv`](./blob/main/eval/benchmark_summary_v592.csv) | per-model aggregates: overall, three family scores, Track/Surface splits, Static and Frozen dimensions |
116
- | [`eval/benchmark_per_case_v592.csv`](./blob/main/eval/benchmark_per_case_v592.csv) | long-form per-case scores (overall + three families) for all 4 models × 200 cases |
117
  | [`eval/significance_tests_v592.json`](./blob/main/eval/significance_tests_v592.json) | paired bootstrap (10k resamples) + Wilcoxon signed-rank for all 6 model pairs |
118
  | [`eval/rubric_stats.json`](./blob/main/eval/rubric_stats.json) | composition of all three rubric families and the aggregation rule |
119
  | [`eval/reports.html`](./blob/main/eval/reports.html) | self-contained interactive HTML report: model comparison, dimension heatmap, score distributions, per-case tables |
120
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
121
  ### Checkpoint selection
122
 
123
  The `data41287` checkpoints were selected by **running the benchmark**, not by taking the
 
112
 
113
  | File | Content |
114
  |---|---|
115
+ | [`eval/per_case_scores.csv`](./blob/main/eval/per_case_scores.csv) | **per-case scores, one row per (model, case)**: 800 rows × overall, three family scores, all six Frozen and all Static dimension scores, frozen passed/total, render_ok |
116
+ | [`eval/per_case_verdicts.jsonl`](./blob/main/eval/per_case_verdicts.jsonl) | **every individual rubric verdict with the judge's evidence sentence** — 800 rows, 51,652 verdicts (Frozen + Prompt Fit + Static), so any score can be traced to a specific criterion |
117
  | [`eval/benchmark_summary_v592.csv`](./blob/main/eval/benchmark_summary_v592.csv) | per-model aggregates: overall, three family scores, Track/Surface splits, Static and Frozen dimensions |
118
+ | [`eval/benchmark_per_case_v592.csv`](./blob/main/eval/benchmark_per_case_v592.csv) | per-case scores in wide form: one row per case, four models side by side |
119
  | [`eval/significance_tests_v592.json`](./blob/main/eval/significance_tests_v592.json) | paired bootstrap (10k resamples) + Wilcoxon signed-rank for all 6 model pairs |
120
  | [`eval/rubric_stats.json`](./blob/main/eval/rubric_stats.json) | composition of all three rubric families and the aggregation rule |
121
  | [`eval/reports.html`](./blob/main/eval/reports.html) | self-contained interactive HTML report: model comparison, dimension heatmap, score distributions, per-case tables |
122
 
123
+ Reading the per-case files:
124
+
125
+ ```python
126
+ import pandas as pd
127
+
128
+ scores = pd.read_csv("eval/per_case_scores.csv")
129
+ scores.pivot_table(index="surface", columns="model", values="overall_score")
130
+
131
+ verdicts = pd.read_json("eval/per_case_verdicts.jsonl", lines=True)
132
+ row = verdicts.query("model == '27B_AdamW_step400'").iloc[0]
133
+ [c for c in row.frozen_checks if c["score"] == 0] # criteria this page failed
134
+ [c for c in row.static_checks if c["score"] == 0] # integrity/craft failures
135
+ ```
136
+
137
  ### Checkpoint selection
138
 
139
  The `data41287` checkpoints were selected by **running the benchmark**, not by taking the