README.md

---
license: cc-by-4.0
language:
- en
base_model:
- FacebookAI/xlm-roberta-large
pipeline_tag: text-classification
library_name: transformers
tags:
- text-classification
- illiberal-policy-frames
- climate
model-index:
- name: poltextlab/illframes-climate-binary-v2
  results:
  - task:
      type: text-classification
      name: Binary illiberal climate-frame detection
    dataset:
      name: ILLFRAMES Climate, held-out split of the training pool
      type: poltextlab/illframes-climate
      split: test
    metrics:
    - name: Accuracy
      type: accuracy
      value: 0.8526
    - name: Macro F1
      type: f1
      args: macro
      value: 0.7462
    - name: Weighted F1
      type: f1
      args: weighted
      value: 0.8508
  - task:
      type: text-classification
      name: Binary illiberal climate-frame detection
    dataset:
      name: ILLFRAMES Climate, SUPERSET_test_data.csv (the climate-v5 test set)
      type: poltextlab/illframes-climate
      split: test_superset
    metrics:
    - name: Accuracy
      type: accuracy
      value: 0.8115
    - name: Macro F1
      type: f1
      args: macro
      value: 0.7766
extra_gated_prompt: This model is not accepting new access requests at the moment.
  Access is still available for our accepted users, which requires the gated access
  setting to stay active. For inquiries or custom project requests, please contact
  us at miklos[dot]sebok[at]poltextlab[dot]com.
extra_gated_fields:
  Country: country
  Institution: text
  Institution Email: text
  Full Name: text
  Please specify your academic project/use case you want to use the models for: text
---

# illframes-climate-binary-v2

**Author:** Miklos Sebok (poltextLAB) — miklos[dot]sebok[at]poltextlab[dot]com

**© 2026 Miklós Sebők.** Released under CC BY 4.0 — reuse requires attribution to the copyright holder.

## Model Description

An **xlm-roberta-large** classifier that decides whether an English sentence carries an illiberal
climate-policy frame. It supersedes `poltextlab/illframes-climate-binary`, whose published metrics were computed on a 969-row
test set that does not exist in the dataset repository and cannot be verified. This release was
trained on a documented split with every evaluation file, prediction file and script included in
the repository, so that each number below can be re-derived.

Labels: **1 = 710–740** (any illiberal climate frame) · **0 = 799** (None of them). The nine-class
codebook is documented on the dataset card
([`poltextlab/illframes-climate`](https://huggingface.co/datasets/poltextlab/illframes-climate)).

Use this model when the quantity of interest is whether a text carries an illiberal climate frame
at all — frame prevalence, diffusion, time series. For the specific frame use `illframes-climate-v5`,
noting that three of its eight frame classes are never recovered (see Limitations).

## How to Use

```python
from transformers import AutoTokenizer, pipeline
tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-large")
pipe = pipeline(model="poltextlab/illframes-climate-binary-v2", task="text-classification", tokenizer=tokenizer,
                use_fast=False, truncation=True, max_length=256, token="<your_hf_read_only_token>")
pipe("<one sentence>")
```

The model requires gated access; pass a token when loading.

## Training data and split

Source: `poltextlab/illframes-climate`. The three non-test files were concatenated
(5177 rows), de-duplicated on normalised text (4740 unique texts; where
duplicates disagreed on the label — 205 groups on the nine-class label,
188 on the binary label — the row from
`TRAINING_DATA_WITHOUT_SUPERSET_training_data.csv` was kept), and then every row that matched
`SUPERSET_test_data.csv` exactly (60) or at TF-IDF cosine ≥ 0.80
(3) was removed, leaving a pool of 4677 rows
(18.3% positive). That pool was split 80/10/10, stratified on the binary label, seed
42: train 3742 · validation 467 · test 468. `SUPERSET_test_data.csv`
(the published test set of `illframes-climate-v5`, 557 rows after removing 3 within-file duplicates)
was held out entirely as a second evaluation set.

Split hygiene was re-derived by an independent check: train, validation and test are pairwise disjoint on
normalised text; no test row has TF-IDF (1–3-gram) cosine ≥ 0.70 to any training row.
The split is at sentence level, not speech level: 97 of 468 test rows (20.7%) share a speech identifier with at least one training row, so the figures below measure generalisation to unseen sentences, not to unseen speeches.

Training-row provenance by identifier prefix: AUT 1486, EP 624, DK 549, DEU 310, HUN 284, USA 195, CZ 177, ES 85, syn 21, en 10, pl 1. Rows prefixed `syn`, `en` and `pl` have no documented origin in the dataset repository and are likely synthetic; they are included in training and disclosed here.

| Setting | Value |
|---|---|
| Base model | `xlm-roberta-large` |
| Learning rate | 8e-06 (warmup 0.1, weight decay 0.01) |
| Epochs | 5 max; early stopping patience 2 on macro-F1; best = epoch 5 |
| Effective batch | 16 × 2 |
| Max length | 256 tokens |
| Precision | fp16 |
| Seed | 42 |

## Performance

### Held-out test (n = 468, 18.2% positive)

accuracy 0.853 · weighted F1 0.851 · macro F1 0.746 · sensitivity 0.565 · specificity 0.916

| Class | Precision | Recall | F1 | Support |
|---|---|---|---|---|
| 0: None of them | 0.905 | 0.916 | 0.911 | 383 |
| 1: Illiberal frame | 0.600 | 0.565 | 0.582 | 85 |

Confusion: TN 351 · FP 32 · FN 37 · TP 48

### `SUPERSET_test_data.csv` — the climate-v5 test set (n = 557, 36.6% positive)

accuracy 0.811 · weighted F1 0.800 · macro F1 0.777 · sensitivity 0.569 · specificity 0.952

| Class | Precision | Recall | F1 | Support |
|---|---|---|---|---|
| 0: None of them | 0.792 | 0.952 | 0.865 | 353 |
| 1: Illiberal frame | 0.872 | 0.569 | 0.688 | 204 |

Confusion: TN 336 · FP 17 · FN 88 · TP 116

### Two operating points

The figures above use the default argmax decision (P(frame) ≥ 0.50). A threshold chosen on the
validation set to maximise macro-F1 gives P(frame) ≥ **0.4**:

| Set | Threshold | Accuracy | Macro F1 | Sensitivity | Specificity |
|---|---|---|---|---|---|
| test | 0.50 | 0.853 | 0.746 | 0.565 | 0.916 |
| test | 0.4 | 0.842 | 0.748 | 0.635 | 0.888 |
| SUPERSET | 0.50 | 0.811 | 0.777 | 0.569 | 0.952 |
| SUPERSET | 0.4 | 0.822 | 0.795 | 0.627 | 0.935 |

`pred__test.csv`, `pred__superset.csv` and `pred__val.csv` in this repository carry P(frame) for every
row, so any other threshold can be evaluated directly.

Validation (n = 467) was used for checkpoint selection and threshold choice and is **not** a
held-out score: accuracy 0.837 · weighted F1 0.835 · macro F1 0.722 · sensitivity 0.523 · specificity 0.908.

### Correcting an observed rate to a prevalence

The share of sentences this model flags is not the share that carry a frame. With sensitivity
0.565 and specificity 0.916 (held-out test, threshold 0.50):

```
p_true = (p_obs + specificity − 1) / (sensitivity + specificity − 1)
```

A corpus with no illiberal framing would still return about 8% positives, and a
corpus that was entirely framed would return about 56%. The denominator
(sensitivity + specificity − 1 = 0.481) is what governs the correction's precision: the smaller it is,
the more an error in either rate is amplified. Report corrected prevalence with an interval, never the raw
positive rate.

## Comparison with the previous models

None of these is a clean head-to-head, and the reasons are stated.

| Model | Scored on | Accuracy | Macro F1 | Sensitivity | Specificity | Why it is not a fair baseline |
|---|---|---|---|---|---|---|
| `poltextlab/illframes-climate-binary` | this held-out test (n = 468) | 0.694 | 0.639 | 0.835 | 0.663 | its training set is undocumented; these test rows come from the same corpus and may have been seen |
| `poltextlab/illframes-climate-binary` | SUPERSET (n = 557) | 0.801 | 0.796 | 0.877 | 0.756 | same |
| `illframes-climate-v5`, nine classes collapsed to binary | SUPERSET (n = 557) | 0.761 | 0.685 | 0.368 | 0.989 | fair for v5 — SUPERSET is its own published test set |

The previous binary model and this one fail in opposite directions: it over-flags (high sensitivity,
low specificity), this one under-flags. The published card of `poltextlab/illframes-climate-binary` (accuracy 0.74, weighted F1 0.76,
n = 969) refers to a test set that no file or combination of files in the dataset repository reproduces.

A second training run on the identical split — class-weighted cross-entropy (weight 4.46 on the frame class), learning rate 1e-05, up to 10 epochs — was tried and rejected: held-out test macro F1 0.724, sensitivity 0.412, specificity 0.966. Its evaluation is kept as `eval_round2.json`.

## Limitations

- **Sensitivity is the weak side.** At the default threshold the model misses 44% of
  framed sentences on the held-out test. The training pool is 18% positive, and the
  first two epochs predicted no frames at all before the model recovered.
- **English, sentence-length input.** Training texts are single sentences (median 26–30 words). Applying
  the model to whole speeches or articles is a unit mismatch — segment first, aggregate after. Use on other
  languages is zero-shot transfer through XLM-R and needs a per-language gold set before the output is
  used substantively.
- **Label contestation in the source data.** 205 texts appear in more than one
  file with different labels; the split resolves them by file priority, not by adjudication.
- **Frame-level resolution is not available for every frame.** On its own published test set,
  `illframes-climate-v5` never recovers classes 721, 722 and 740 (F1 0.00).
  This binary model does not distinguish frames, so that gap does not affect it, but users who need the
  specific frame should know it exists.

## Files

`model.safetensors` · tokenizer · `config.json` · `eval.json` (all metrics) · `thresholds.json`
(validation sweep + applied thresholds) · `pred__*.csv` (per-row P(frame)) · `eval_round2.json`
(the rejected second run) · `scripts/` (split, training, baseline, verification and card scripts).

## Cooperation

Model performance can be improved by extending the training sets. Coded corpora of any domain and
language are welcome at poltextlab{at}poltextlab{dot}com.