TianmaPegasus commited on
Commit
ec51d3b
Β·
verified Β·
1 Parent(s): 623735d

Replace repo with official FlexICM checkpoints only

Browse files

Remove previous code/weights and publish the mentor-selected TAIC/C-TAIC checkpoint_best_loss set under checkpoints/.

This view is limited to 50 files because it contains too many changes. Β  See raw diff
Files changed (50) hide show
  1. .gitignore +0 -32
  2. README.md +1 -435
  3. checkpoints/README.md +0 -92
  4. checkpoints/base_codec/NOTE.txt +0 -2
  5. checkpoints/base_codec/PLACEHOLDER_base_codec_1.pth.tar.txt +0 -2
  6. checkpoints/base_codec/PLACEHOLDER_base_codec_2.pth.tar.txt +0 -2
  7. checkpoints/base_codec/PLACEHOLDER_base_codec_3.pth.tar.txt +0 -2
  8. checkpoints/base_codec/PLACEHOLDER_base_codec_4.pth.tar.txt +0 -2
  9. checkpoints/ctaic/s1_det_instance/stage1/1/PLACEHOLDER +0 -2
  10. checkpoints/ctaic/s1_det_instance/stage1/1/{checkpoint_0.1313.pth.tar β†’ checkpoint_best_loss.pth.tar} +0 -0
  11. checkpoints/ctaic/s1_det_instance/stage1/2/PLACEHOLDER +0 -2
  12. checkpoints/ctaic/s1_det_instance/stage1/2/{checkpoint_0.1334.pth.tar β†’ checkpoint_best_loss.pth.tar} +0 -0
  13. checkpoints/ctaic/s1_det_instance/stage1/3/PLACEHOLDER +0 -2
  14. checkpoints/ctaic/s1_det_instance/stage1/3/{checkpoint_0.1564.pth.tar β†’ checkpoint_best_loss.pth.tar} +0 -0
  15. checkpoints/ctaic/s1_det_instance/stage1/4/PLACEHOLDER +0 -2
  16. checkpoints/ctaic/s1_det_instance/stage1/4/{checkpoint_0.1245.pth.tar β†’ checkpoint_best_loss.pth.tar} +0 -0
  17. checkpoints/ctaic/s1_det_instance/stage2/1/PLACEHOLDER +0 -2
  18. checkpoints/ctaic/{s2_sem_panoptic/stage1/1/checkpoint_0.1313.pth.tar β†’ s1_det_instance/stage2/1/checkpoint_best_loss.pth.tar} +0 -0
  19. checkpoints/ctaic/s1_det_instance/stage2/2/PLACEHOLDER +0 -2
  20. checkpoints/ctaic/{s2_sem_panoptic/stage1/2/checkpoint_0.1334.pth.tar β†’ s1_det_instance/stage2/2/checkpoint_best_loss.pth.tar} +0 -0
  21. checkpoints/ctaic/s1_det_instance/stage2/3/PLACEHOLDER +0 -2
  22. checkpoints/ctaic/{s2_sem_panoptic/stage1/3/checkpoint_0.1564.pth.tar β†’ s1_det_instance/stage2/3/checkpoint_best_loss.pth.tar} +0 -0
  23. checkpoints/ctaic/s1_det_instance/stage2/4/PLACEHOLDER +0 -2
  24. checkpoints/ctaic/{s2_sem_panoptic/stage1/4/checkpoint_0.1245.pth.tar β†’ s1_det_instance/stage2/4/checkpoint_best_loss.pth.tar} +0 -0
  25. checkpoints/ctaic/s2_sem_panoptic/stage1/1/PLACEHOLDER +0 -2
  26. checkpoints/ctaic/{s3_det_pose/stage1/1/checkpoint_0.1313.pth.tar β†’ s2_sem_panoptic/stage1/1/checkpoint_best_loss.pth.tar} +0 -0
  27. checkpoints/ctaic/s2_sem_panoptic/stage1/2/PLACEHOLDER +0 -2
  28. checkpoints/ctaic/{s3_det_pose/stage1/2/checkpoint_0.1334.pth.tar β†’ s2_sem_panoptic/stage1/2/checkpoint_best_loss.pth.tar} +0 -0
  29. checkpoints/ctaic/s2_sem_panoptic/stage1/3/PLACEHOLDER +0 -2
  30. checkpoints/ctaic/{s3_det_pose/stage1/3/checkpoint_0.1564.pth.tar β†’ s2_sem_panoptic/stage1/3/checkpoint_best_loss.pth.tar} +0 -0
  31. checkpoints/ctaic/s2_sem_panoptic/stage1/4/PLACEHOLDER +0 -2
  32. checkpoints/ctaic/{s3_det_pose/stage1/4/checkpoint_0.1245.pth.tar β†’ s2_sem_panoptic/stage1/4/checkpoint_best_loss.pth.tar} +0 -0
  33. checkpoints/ctaic/s2_sem_panoptic/stage2/1/PLACEHOLDER +0 -2
  34. checkpoints/{taic/detection/1/checkpoint_0.1313.pth.tar β†’ ctaic/s2_sem_panoptic/stage2/1/checkpoint_best_loss.pth.tar} +0 -0
  35. checkpoints/ctaic/s2_sem_panoptic/stage2/2/PLACEHOLDER +0 -2
  36. checkpoints/{taic/detection/2/checkpoint_0.1334.pth.tar β†’ ctaic/s2_sem_panoptic/stage2/2/checkpoint_best_loss.pth.tar} +0 -0
  37. checkpoints/ctaic/s2_sem_panoptic/stage2/3/PLACEHOLDER +0 -2
  38. checkpoints/{taic/detection/3/checkpoint_0.1564.pth.tar β†’ ctaic/s2_sem_panoptic/stage2/3/checkpoint_best_loss.pth.tar} +0 -0
  39. checkpoints/ctaic/s2_sem_panoptic/stage2/4/PLACEHOLDER +0 -2
  40. checkpoints/{taic/detection/4/checkpoint_0.1245.pth.tar β†’ ctaic/s2_sem_panoptic/stage2/4/checkpoint_best_loss.pth.tar} +0 -0
  41. checkpoints/ctaic/s3_det_pose/stage1/1/PLACEHOLDER +0 -2
  42. checkpoints/{taic/instance/1/checkpoint_0.1313.pth.tar β†’ ctaic/s3_det_pose/stage1/1/checkpoint_best_loss.pth.tar} +0 -0
  43. checkpoints/ctaic/s3_det_pose/stage1/2/PLACEHOLDER +0 -2
  44. checkpoints/{taic/instance/2/checkpoint_0.1334.pth.tar β†’ ctaic/s3_det_pose/stage1/2/checkpoint_best_loss.pth.tar} +0 -0
  45. checkpoints/ctaic/s3_det_pose/stage1/3/PLACEHOLDER +0 -2
  46. checkpoints/{taic/instance/3/checkpoint_0.1564.pth.tar β†’ ctaic/s3_det_pose/stage1/3/checkpoint_best_loss.pth.tar} +0 -0
  47. checkpoints/ctaic/s3_det_pose/stage1/4/PLACEHOLDER +0 -2
  48. checkpoints/{taic/instance/4/checkpoint_0.1245.pth.tar β†’ ctaic/s3_det_pose/stage1/4/checkpoint_best_loss.pth.tar} +0 -0
  49. checkpoints/ctaic/s3_det_pose/stage2/1/PLACEHOLDER +0 -2
  50. checkpoints/ctaic/s3_det_pose/stage2/1/checkpoint_best_loss.pth.tar +3 -0
.gitignore DELETED
@@ -1,32 +0,0 @@
1
- __pycache__/
2
- *.py[cod]
3
- *.egg-info/
4
- .eggs/
5
- dist/
6
- build/
7
- .DS_Store
8
- logs/
9
- *.log
10
- coco_log/
11
- .ipynb_checkpoints/
12
- _figs/
13
- _paper_extract.txt
14
-
15
- # paper PDF β€” do not upload
16
- FlexICM.pdf
17
- *.pdf
18
-
19
- # Heavy weights to keep local-only.
20
- # TAIC / C-TAIC under checkpoints/{taic,ctaic} are intentionally uploadable
21
- # (avoid broad checkpoints/** + !negation β€” Hub upload handles ! poorly).
22
- checkpoints/base_codec/**/*.pth.tar
23
- checkpoints/base_codec/**/*.tar
24
- checkpoints/base_codec/**/*.pkl
25
- checkpoints/base_codec/**/*.pth
26
- checkpoints/task_networks/**/*.pth.tar
27
- checkpoints/task_networks/**/*.tar
28
- checkpoints/task_networks/**/*.pkl
29
- checkpoints/task_networks/**/*.pth
30
- !checkpoints/**/PLACEHOLDER*
31
- !checkpoints/**/*.txt
32
- !checkpoints/README.md
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
README.md CHANGED
@@ -8,439 +8,5 @@ tags:
8
 
9
  # FlexICM: A Flexible Image Coding for Machines Framework
10
 
11
- Official codebase for the paper **FlexICM: A Flexible Image Coding for Machines Framework** (Tianma Shen, Ying Liu).
12
-
13
- Built on the **TIC (Transformer-based Image Compression)** base codec, this repository implements:
14
-
15
- - **TAIC (Base Layer)**: five single-task codecs that decode task intermediate features `h` **without** full image reconstruction
16
- - **C-TAIC (Extension Layer)**: three multi-task scenarios that condition on the base-layer latent \hat{y}_b via cross-attention
17
-
18
- ## Five Tasks and Three Scenarios
19
-
20
- ### TAIC (five task codecs)
21
-
22
-
23
- | Task | Teacher / Task Network | Feature Alignment | Metric |
24
- | --------------------- | ------------------------------- | ----------------------- | -------- |
25
- | Object Detection | Faster R-CNN + **Swin-B** | FPN `P2..P6` (Eq. 2) | mAP-bbox |
26
- | Semantic Segmentation | UPerNet + **Swin-B** | FPN `P2..P6` | mIoU |
27
- | Instance Segmentation | Cascade Mask R-CNN + **Swin-B** | FPN `P2..P6` | mAP-mask |
28
- | Panoptic Segmentation | MaskFormer + **Swin-B** | Stages `F1..F4` (Eq. 3) | PQ |
29
- | Pose Estimation | **HigherHRNet** | Stages `F1..F4` | mAP-OKS |
30
-
31
-
32
-
33
-
34
- ### C-TAIC (three scenarios)
35
-
36
-
37
- | Scenario | Base Layer | Extension Layer |
38
- | -------- | --------------------- | --------------------- |
39
- | **s1** | Object Detection | Instance Segmentation |
40
- | **s2** | Semantic Segmentation | Panoptic Segmentation |
41
- | **s3** | Object Detection | Pose Estimation |
42
-
43
-
44
- ---
45
-
46
-
47
-
48
- ## Environment Setup
49
-
50
- > **Important:** Codec training **requires** task networks (teachers) to be available.
51
- > The loss D is computed from frozen teacher features, so you cannot train TAIC / C-TAIC
52
- > with only the codec packages. Install the teacher stack in **Task networks (teachers)** before the first training run.
53
-
54
-
55
-
56
- ### Recommended environment
57
-
58
- - Ubuntu / RHEL, **CUDA 11.7+**, single **NVIDIA A100** (paper setting)
59
- - Python **3.8–3.10**
60
- - PyTorch **β‰₯ 1.12** (2.0+ recommended)
61
-
62
- ```bash
63
- conda create -n flexicm python=3.9 -y
64
- conda activate flexicm
65
- pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118
66
- pip install -r requirements.txt
67
- ```
68
-
69
-
70
-
71
- ### Core codec dependencies
72
-
73
-
74
- | Package | Role |
75
- | ------------ | ---------------------------------------------------------- |
76
- | `compressai` | EntropyBottleneck / GaussianConditional / conv-deconv |
77
- | `timm` | **Required** Swin-B teacher backbone for feature alignment |
78
- | `PyYAML` | Training configs |
79
-
80
-
81
-
82
-
83
- ### Task networks (teachers) β€” **required before training**
84
-
85
- Teachers are already implemented in `flexicm/tasks/` and are constructed automatically by
86
- `scripts/train_taic.py` / `scripts/train_ctaic.py` via `build_teacher(...)`.
87
- You still must install their runtime dependencies and allow pretrained weights to download.
88
-
89
-
90
- | Task | Teacher used in training | What you need installed |
91
- | -------------------- | ---------------------------------------------------------------------------- | ---------------------------------------------------- |
92
- | Detection / Instance | Swin-B (`timm`) for FPN alignment; metric head = Cascade Mask R-CNN + Swin-B | `timm`; ImageNet Swin-B on first run |
93
- | Semantic / Panoptic | Swin-B (`timm`) | `timm`; ImageNet Swin-B on first run |
94
- | Pose | HigherHRNet-style HRNet stem (original HRNet, not Swin) | Implemented in-repo; no extra package beyond PyTorch |
95
-
96
-
97
- Checklist before training:
98
-
99
- 1. `pip install -r requirements.txt` (includes `timm`)
100
- 2. Machine can reach the internet **or** you have cached `timm` Swin weights
101
- 3. Verify teachers import cleanly:
102
-
103
- ```bash
104
- python -c "from flexicm.tasks import build_teacher; build_teacher('detection'); print('teachers ok')"
105
- ```
106
-
107
- Without a working teacher, training will fail when computing the feature-alignment term D.
108
-
109
- ### Task heads for metric evaluation
110
-
111
- To evaluate paper metrics (mAP / mIoU / PQ / OKS) with full task heads, also install:
112
-
113
- ```bash
114
- pip install -U openmim
115
- mim install mmengine mmcv
116
- mim install mmdet mmsegmentation mmpose
117
- ```
118
-
119
- Official detection / instance weights from
120
- [Swin-Transformer-Object-Detection](https://github.com/SwinTransformer/Swin-Transformer-Object-Detection)
121
- (see `configs/task_networks/README.md`):
122
-
123
- - **Cascade Mask R-CNN + Swin-B** (detection mAP-bbox **and** instance mAP-mask; same weights)
124
- - **UPerNet + Swin-B**: MMSegmentation Model Zoo
125
- - **MaskFormer + Swin-B**: MMDetection / Mask2Former
126
- - **HigherHRNet**: MMPose Model Zoo (**HRNet backbone**)
127
-
128
- These full heads are **not** required to start codec training; they are for final rate–accuracy evaluation.
129
-
130
- ---
131
-
132
-
133
-
134
- ## Repository Layout
135
-
136
- ```
137
- FlexICM/
138
- β”œβ”€β”€ FlexICM.pdf # paper
139
- β”œβ”€β”€ requirements.txt
140
- β”œβ”€β”€ README.md
141
- β”œβ”€β”€ configs/
142
- β”‚ β”œβ”€β”€ taic/ # five single-task configs
143
- β”‚ β”‚ β”œβ”€β”€ detection.yaml
144
- β”‚ β”‚ β”œβ”€β”€ semantic.yaml
145
- β”‚ β”‚ β”œβ”€β”€ instance.yaml
146
- β”‚ β”‚ β”œβ”€β”€ panoptic.yaml
147
- β”‚ β”‚ └── pose.yaml
148
- β”‚ └── ctaic/ # three multi-task scenarios
149
- β”‚ β”œβ”€β”€ s1_det_instance.yaml
150
- β”‚ β”œβ”€β”€ s2_sem_panoptic.yaml
151
- β”‚ └── s3_det_pose.yaml
152
- β”œβ”€β”€ scripts/
153
- β”‚ β”œβ”€β”€ download_base_codecs.sh
154
- β”‚ β”œβ”€β”€ train_taic.py
155
- β”‚ β”œβ”€β”€ train_ctaic.py
156
- β”‚ β”œβ”€β”€ eval_taic.py # codec test (bpp / feature D)
157
- β”‚ └── eval_ctaic.py # codec test for C-TAIC
158
- β”œβ”€β”€ flexicm/
159
- β”‚ β”œβ”€β”€ models/ # TAIC / C-TAIC / SFMA / TaskConnector / Conditional
160
- β”‚ β”œβ”€β”€ layers/ # RSTB / WindowAttention (same lineage as AdaptiveICMH)
161
- β”‚ β”œβ”€β”€ tasks/ # teachers + feature-alignment losses
162
- β”‚ β”œβ”€β”€ data/ # COCO / COCO-WholeBody image loading
163
- β”‚ └── utils/
164
- └── checkpoints/ # placeholder tree for base / TAIC / C-TAIC weights
165
- # see checkpoints/README.md
166
- ```
167
-
168
- Eval / codec-test configs: `configs/eval/`.
169
-
170
- ---
171
-
172
-
173
-
174
- ## Dataset Preparation
175
-
176
-
177
-
178
- ### COCO-2017 (detection / instance / semantic / panoptic)
179
-
180
- ```text
181
- /data/coco2017/
182
- β”œβ”€β”€ train2017/
183
- β”œβ”€β”€ val2017/
184
- └── annotations/
185
- β”œβ”€β”€ instances_train2017.json
186
- β”œβ”€β”€ instances_val2017.json
187
- β”œβ”€β”€ panoptic_train2017.json
188
- β”œβ”€β”€ panoptic_val2017.json
189
- β”œβ”€β”€ panoptic_train2017/ # PNG
190
- β”œβ”€β”€ panoptic_val2017/
191
- β”œβ”€β”€ stuff_train2017.json # semantic / stuff (if used)
192
- └── stuff_val2017.json
193
- ```
194
-
195
- Download:
196
-
197
- ```bash
198
- # images
199
- wget http://images.cocodataset.org/zips/train2017.zip
200
- wget http://images.cocodataset.org/zips/val2017.zip
201
- # annotations
202
- wget http://images.cocodataset.org/annotations/annotations_trainval2017.zip
203
- wget http://images.cocodataset.org/annotations/panoptic_annotations_trainval2017.zip
204
- ```
205
-
206
- Set in the corresponding YAML:
207
-
208
- ```yaml
209
- dataset_path: "/data/coco2017"
210
- ```
211
-
212
-
213
-
214
- ### COCO-WholeBody (pose estimation)
215
-
216
- Pose uses the same COCO `train2017/val2017` images plus WholeBody keypoint annotations:
217
-
218
- - Project page: [COCO-WholeBody](https://github.com/jin-s13/COCO-WholeBody)
219
- - Place JSON files under `annotations/`; evaluate with MMPose HigherHRNet + WholeBody configs
220
-
221
- Codec **training** only needs images for feature alignment, so `train2017` images are sufficient for that stage.
222
-
223
- ### Data processing (aligned with task networks)
224
-
225
- The paper requires **codec training preprocessing to match task-network preprocessing**. Defaults in this repo:
226
-
227
- 1. **Codec input**: RGB, `ToTensor()` β†’ `[0,1]`; training uses `Resize β†’ RandomCrop(256) β†’ RandomHorizontalFlip`
228
- 2. **Inside the teacher**: ImageNet mean/std normalization (consistent with Swin / HRNet pretraining)
229
- 3. **Spatial alignment**: TIC requires spatial size divisible by **256** (256 crop for training; pad at inference)
230
-
231
- If you use official MMDet/MMSeg pipelines (short-side resize, normalization, etc.), ensure:
232
-
233
- - Teacher feature extraction uses the **same normalize / resize logic** as that task network
234
- - Codec and teacher see geometrically consistent tensors (same crop / same pad)
235
-
236
- Edit points: `flexicm/data/datasets.py`, `flexicm/tasks/swin_teacher.py`, `flexicm/tasks/__init__.py` (HigherHRNet).
237
-
238
- ---
239
-
240
-
241
-
242
- ## Base Codec (TIC) Checkpoints
243
-
244
- The paper uses the same TIC pretrained weights as AdaptiveICMH / TransTIC:
245
-
246
-
247
- | Quality | Ξ» (paper) | Checkpoint |
248
- | ------- | --------- | ------------------------------------------------------------------------------------------------- |
249
- | 1 | 0.0035 | [base_codec_1](https://github.com/NYCU-MAPL/TransTIC/releases/download/v1.0/base_codec_1.pth.tar) |
250
- | 2 | 0.0067 | [base_codec_2](https://github.com/NYCU-MAPL/TransTIC/releases/download/v1.0/base_codec_2.pth.tar) |
251
- | 3 | 0.0130 | [base_codec_3](https://github.com/NYCU-MAPL/TransTIC/releases/download/v1.0/base_codec_3.pth.tar) |
252
- | 4 | 0.0250 | [base_codec_4](https://github.com/NYCU-MAPL/TransTIC/releases/download/v1.0/base_codec_4.pth.tar) |
253
-
254
-
255
- ```bash
256
- bash scripts/download_base_codecs.sh
257
- # downloads into checkpoints/base_codec/base_codec_{1,2,3,4}.pth.tar
258
- ```
259
-
260
- Config example:
261
-
262
- ```yaml
263
- base_codec: "./checkpoints/base_codec/base_codec_1.pth.tar"
264
- quality_level: 1
265
- lmbda: 0.0035
266
- ```
267
-
268
- For each bitrate point, switch the matching `base_codec_k` and `lmbda`.
269
-
270
- Trained TAIC / C-TAIC weights for eval should be placed under `checkpoints/taic/` and
271
- `checkpoints/ctaic/` (see `checkpoints/README.md`). Until then, each quality folder
272
- contains a `PLACEHOLDER` file.
273
-
274
- ---
275
-
276
-
277
-
278
- ## Training
279
-
280
- Paper settings:
281
-
282
- - Optimizer: **AdamW**, `lr=1e-4`
283
- - TAIC: `batch_size=80`, `epochs=35`
284
- - C-TAIC: `batch_size=40`, `epochs=40`
285
- - `λ ∈ {0.0035, 0.0067, 0.0130, 0.0250}`
286
-
287
- > If GPU memory is insufficient, reduce `batch_size` (optionally use gradient accumulation to approximate the paper effective batch).
288
-
289
-
290
-
291
- ### Train five TAIC models
292
-
293
- ```bash
294
- # edit dataset_path / base_codec / lmbda / gpu_id in configs/taic/*.yaml as needed
295
- python scripts/train_taic.py -c configs/taic/detection.yaml
296
- python scripts/train_taic.py -c configs/taic/semantic.yaml
297
- python scripts/train_taic.py -c configs/taic/instance.yaml
298
- python scripts/train_taic.py -c configs/taic/panoptic.yaml
299
- python scripts/train_taic.py -c configs/taic/pose.yaml
300
- ```
301
-
302
- Trainable modules: **encoder SFMA + Task Connector**; TIC trunk is frozen.
303
-
304
- ### Train three C-TAIC scenarios
305
-
306
- Requires a trained **base TAIC** checkpoint (to provide \hat{y}_b) and Stage-1 weights for the extension task.
307
-
308
- ```bash
309
- # ---- s1: det β†’ instance ----
310
- python scripts/train_ctaic.py -c configs/ctaic/s1_det_instance.yaml --stage 1
311
- python scripts/train_ctaic.py -c configs/ctaic/s1_det_instance.yaml --stage 2
312
-
313
- # ---- s2: semantic β†’ panoptic ----
314
- python scripts/train_ctaic.py -c configs/ctaic/s2_sem_panoptic.yaml --stage 1
315
- python scripts/train_ctaic.py -c configs/ctaic/s2_sem_panoptic.yaml --stage 2
316
-
317
- # ---- s3: det β†’ pose ----
318
- python scripts/train_ctaic.py -c configs/ctaic/s3_det_pose.yaml --stage 1
319
- python scripts/train_ctaic.py -c configs/ctaic/s3_det_pose.yaml --stage 2
320
- ```
321
-
322
- Stage meanings:
323
-
324
-
325
- | Stage | Mode | Trainable modules | `Ε·_b` |
326
- | ----- | ----------- | -------------------------------------- | ------------------------------- |
327
- | 1 | TAIC mode | SFMA + Task Connector | not used |
328
- | 2 | C-TAIC mode | Prompt Generator + Condition Generator | from frozen base TAIC AD output |
329
-
330
-
331
- Check these config fields:
332
-
333
- ```yaml
334
- base_taic_checkpoint: # trained TAIC for the base task
335
- taic_init: # extension-task TAIC (optional Stage-1 init)
336
- stage1_checkpoint: # Stage-1 result loaded in Stage 2
337
- ```
338
-
339
- ---
340
-
341
-
342
-
343
- ## Codec Test
344
-
345
- Codec test measures **compression statistics**:
346
-
347
-
348
- | Metric | Meaning |
349
- | ------------ | ----------------------------------------------------------------- |
350
- | `bpp` | Likelihood bitrate R |
351
- | `distortion` | Feature alignment D (Eq. 2 or Eq. 3) |
352
- | `loss` | R + \lambda D |
353
- | `actual_bpp` | Optional: real bitstream size after `compress()` / `decompress()` |
354
-
355
-
356
- For C-TAIC, reported `bpp` is **extension-layer only** (base-layer rate is excluded), matching the paper.
357
-
358
- ### Prepare codec checkpoints
359
-
360
- 1. Copy trained weights into `checkpoints/taic/` or `checkpoints/ctaic/` (see `checkpoints/README.md`)
361
- 2. Remove the local `PLACEHOLDER` once `checkpoint_best_loss.pth.tar` is present
362
- 3. Edit `dataset_path` / `gpu_id` in `configs/eval/*.yaml`
363
-
364
- ```bash
365
- python scripts/eval_taic.py -c configs/eval/taic_detection.yaml
366
- python scripts/eval_ctaic.py -c configs/eval/ctaic_s1.yaml
367
- ```
368
-
369
- ---
370
-
371
-
372
-
373
- ## Task-network metric evaluation
374
-
375
- To reproduce paper rate–accuracy numbers you must **also** load the official pretrained
376
- **task networks** and run metrics on COCO val:
377
-
378
-
379
- | Task | Task network | Metric |
380
- | --------- | ---------------------------- | -------- |
381
- | Detection | Cascade Mask R-CNN + Swin-B | mAP-bbox |
382
- | Instance | Cascade Mask R-CNN + Swin-B | mAP-mask |
383
- | Semantic | UPerNet + Swin-B | mIoU |
384
- | Panoptic | MaskFormer + Swin-B | PQ |
385
- | Pose | HigherHRNet (HRNet backbone) | mAP-OKS |
386
-
387
-
388
- Pipeline: `image β†’ codec β†’ h β†’ truncated task net (from Stage2 / FPN) β†’ metric`.
389
-
390
- ### Install metric dependencies
391
-
392
- ```bash
393
- pip install pycocotools
394
- pip install -U openmim
395
- mim install mmengine mmcv mmdet mmsegmentation mmpose
396
- # optional for PQ:
397
- # pip install git+https://github.com/cocodataset/panopticapi.git
398
- ```
399
-
400
-
401
-
402
- ### Prepare task-network configs & checkpoints
403
-
404
- 1. Put / symlink real OpenMMLab configs under `configs/task_networks/`
405
- (see `configs/task_networks/README.md`; current `*.py` files are stubs)
406
- 2. Download official weights to:
407
-
408
- ```text
409
- checkpoints/task_networks/
410
- β”œβ”€β”€ detection/model.pth
411
- β”œβ”€β”€ instance/model.pth
412
- β”œβ”€β”€ semantic/model.pth
413
- β”œβ”€β”€ panoptic/model.pth
414
- └── pose/model.pth
415
- ```
416
-
417
- 1. Set in each `configs/eval/*.yaml`:
418
-
419
- ```yaml
420
- task_config: "./configs/task_networks/<real_config>.py"
421
- task_checkpoint: "./checkpoints/task_networks/<task>/model.pth"
422
- ann_file: "annotations/instances_val2017.json"
423
- ```
424
-
425
-
426
-
427
- ### Run codec + metrics
428
-
429
- ```bash
430
- python scripts/eval_taic.py -c configs/eval/taic_detection.yaml
431
- python scripts/eval_taic.py -c configs/eval/taic_instance.yaml
432
- python scripts/eval_taic.py -c configs/eval/taic_semantic.yaml
433
- python scripts/eval_taic.py -c configs/eval/taic_panoptic.yaml
434
- python scripts/eval_taic.py -c configs/eval/taic_pose.yaml
435
-
436
- python scripts/eval_ctaic.py -c configs/eval/ctaic_s1.yaml
437
- python scripts/eval_ctaic.py -c configs/eval/ctaic_s2.yaml
438
- python scripts/eval_ctaic.py -c configs/eval/ctaic_s3.yaml
439
- ```
440
-
441
- JSON results (codec + task metrics) are written under `logs/eval_taic/` or `logs/eval_ctaic/`.
442
-
443
- > Detection / instance metric paths are the most complete (COCO bbox via pycocotools).
444
- > Semantic mIoU needs a GT label loader; panoptic PQ needs `panopticapi` + GT folders;
445
- > pose-from-`h` may need a HigherHRNet stem hook for your exact MMPose version.
446
 
 
8
 
9
  # FlexICM: A Flexible Image Coding for Machines Framework
10
 
11
+ Official checkpoints for the paper **FlexICM: A Flexible Image Coding for Machines Framework** (Tianma Shen, Ying Liu).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
12
 
checkpoints/README.md DELETED
@@ -1,92 +0,0 @@
1
- # Checkpoints Layout (Placeholders)
2
-
3
- Put pretrained / trained weights here before running **test / eval**.
4
- Training still writes to `logs/` by default; after training, copy (or symlink) best
5
- checkpoints into this tree so eval configs have a stable path.
6
-
7
- > Files named `PLACEHOLDER` are not real weights. Replace each with the matching
8
- > `.pth.tar` checkpoint, then update or keep the path expected by eval scripts.
9
-
10
- ## Directory map
11
-
12
- ```text
13
- checkpoints/
14
- β”œβ”€β”€ base_codec/ # frozen TIC (TransTIC / AdaptiveICMH)
15
- β”‚ β”œβ”€β”€ base_codec_1.pth.tar # Ξ» = 0.0035
16
- β”‚ β”œβ”€β”€ base_codec_2.pth.tar # Ξ» = 0.0067
17
- β”‚ β”œβ”€β”€ base_codec_3.pth.tar # Ξ» = 0.0130
18
- β”‚ └── base_codec_4.pth.tar # Ξ» = 0.0250
19
- β”‚
20
- β”œβ”€β”€ taic/ # five single-task TAIC codecs
21
- β”‚ β”œβ”€β”€ detection/{1,2,3,4}/checkpoint_best_loss.pth.tar
22
- β”‚ β”œβ”€β”€ semantic/{1,2,3,4}/checkpoint_best_loss.pth.tar
23
- β”‚ β”œβ”€β”€ instance/{1,2,3,4}/checkpoint_best_loss.pth.tar
24
- β”‚ β”œβ”€β”€ panoptic/{1,2,3,4}/checkpoint_best_loss.pth.tar
25
- β”‚ └── pose/{1,2,3,4}/checkpoint_best_loss.pth.tar
26
- β”‚
27
- └── ctaic/ # three multi-task scenarios
28
- β”œβ”€β”€ s1_det_instance/
29
- β”‚ β”œβ”€β”€ stage1/{1,2,3,4}/checkpoint_best_loss.pth.tar
30
- β”‚ └── stage2/{1,2,3,4}/checkpoint_best_loss.pth.tar
31
- β”œβ”€β”€ s2_sem_panoptic/
32
- β”‚ β”œβ”€β”€ stage1/{1,2,3,4}/checkpoint_best_loss.pth.tar
33
- β”‚ └── stage2/{1,2,3,4}/checkpoint_best_loss.pth.tar
34
- └── s3_det_pose/
35
- β”œβ”€β”€ stage1/{1,2,3,4}/checkpoint_best_loss.pth.tar
36
- └── stage2/{1,2,3,4}/checkpoint_best_loss.pth.tar
37
- ```
38
-
39
- Quality folders `{1,2,3,4}` match paper Ξ» / TIC quality levels.
40
-
41
- ## Download base TIC codecs
42
-
43
- ```bash
44
- bash scripts/download_base_codecs.sh
45
- # downloads into checkpoints/base_codec/
46
- ```
47
-
48
- ## After training: copy into placeholders
49
-
50
- ```bash
51
- # example: TAIC detection, quality 1
52
- cp logs/taic_detection/1/checkpoint_best_loss.pth.tar \
53
- checkpoints/taic/detection/1/checkpoint_best_loss.pth.tar
54
-
55
- # example: C-TAIC s1 stage2, quality 1
56
- cp logs/ctaic_s1_stage2/1/checkpoint_best_loss.pth.tar \
57
- checkpoints/ctaic/s1_det_instance/stage2/1/checkpoint_best_loss.pth.tar
58
- ```
59
-
60
- ## Eval configs
61
-
62
- See `configs/eval/` β€” they point to these placeholder paths.
63
-
64
- Codec test (bpp / feature distortion):
65
-
66
- ```bash
67
- python scripts/eval_taic.py -c configs/eval/taic_detection.yaml
68
- python scripts/eval_ctaic.py -c configs/eval/ctaic_s1.yaml
69
- ```
70
-
71
- Scripts refuse to run if a `PLACEHOLDER` file is still present or if the `.pth.tar` is missing.
72
- Task rate–accuracy metrics are not implemented in these scripts yet.
73
-
74
- ## Task networks (for metric evaluation)
75
-
76
- ```text
77
- checkpoints/task_networks/
78
- β”œβ”€β”€ detection/model.pth
79
- β”œβ”€β”€ instance/model.pth
80
- β”œβ”€β”€ semantic/model.pth
81
- β”œβ”€β”€ panoptic/model.pth
82
- └── pose/model.pth
83
- ```
84
-
85
- These are **official pretrained task networks** (not codec weights).
86
- Required when running:
87
-
88
- ```bash
89
- python scripts/eval_taic.py -c configs/eval/taic_detection.yaml --with-metrics
90
- ```
91
-
92
- See `configs/task_networks/README.md` for config/checkpoint pairing.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
checkpoints/base_codec/NOTE.txt DELETED
@@ -1,2 +0,0 @@
1
- Legacy config paths used ./checkpoints/base_codec_k.pth.tar.
2
- Prefer checkpoints/base_codec/base_codec_k.pth.tar after download.
 
 
 
checkpoints/base_codec/PLACEHOLDER_base_codec_1.pth.tar.txt DELETED
@@ -1,2 +0,0 @@
1
- PLACEHOLDER: replace this file with the real checkpoint (.pth.tar).
2
- See checkpoints/README.md for the expected filename and training copy commands.
 
 
 
checkpoints/base_codec/PLACEHOLDER_base_codec_2.pth.tar.txt DELETED
@@ -1,2 +0,0 @@
1
- PLACEHOLDER: replace this file with the real checkpoint (.pth.tar).
2
- See checkpoints/README.md for the expected filename and training copy commands.
 
 
 
checkpoints/base_codec/PLACEHOLDER_base_codec_3.pth.tar.txt DELETED
@@ -1,2 +0,0 @@
1
- PLACEHOLDER: replace this file with the real checkpoint (.pth.tar).
2
- See checkpoints/README.md for the expected filename and training copy commands.
 
 
 
checkpoints/base_codec/PLACEHOLDER_base_codec_4.pth.tar.txt DELETED
@@ -1,2 +0,0 @@
1
- PLACEHOLDER: replace this file with the real checkpoint (.pth.tar).
2
- See checkpoints/README.md for the expected filename and training copy commands.
 
 
 
checkpoints/ctaic/s1_det_instance/stage1/1/PLACEHOLDER DELETED
@@ -1,2 +0,0 @@
1
- PLACEHOLDER: replace this file with the real checkpoint (.pth.tar).
2
- See checkpoints/README.md for the expected filename and training copy commands.
 
 
 
checkpoints/ctaic/s1_det_instance/stage1/1/{checkpoint_0.1313.pth.tar β†’ checkpoint_best_loss.pth.tar} RENAMED
File without changes
checkpoints/ctaic/s1_det_instance/stage1/2/PLACEHOLDER DELETED
@@ -1,2 +0,0 @@
1
- PLACEHOLDER: replace this file with the real checkpoint (.pth.tar).
2
- See checkpoints/README.md for the expected filename and training copy commands.
 
 
 
checkpoints/ctaic/s1_det_instance/stage1/2/{checkpoint_0.1334.pth.tar β†’ checkpoint_best_loss.pth.tar} RENAMED
File without changes
checkpoints/ctaic/s1_det_instance/stage1/3/PLACEHOLDER DELETED
@@ -1,2 +0,0 @@
1
- PLACEHOLDER: replace this file with the real checkpoint (.pth.tar).
2
- See checkpoints/README.md for the expected filename and training copy commands.
 
 
 
checkpoints/ctaic/s1_det_instance/stage1/3/{checkpoint_0.1564.pth.tar β†’ checkpoint_best_loss.pth.tar} RENAMED
File without changes
checkpoints/ctaic/s1_det_instance/stage1/4/PLACEHOLDER DELETED
@@ -1,2 +0,0 @@
1
- PLACEHOLDER: replace this file with the real checkpoint (.pth.tar).
2
- See checkpoints/README.md for the expected filename and training copy commands.
 
 
 
checkpoints/ctaic/s1_det_instance/stage1/4/{checkpoint_0.1245.pth.tar β†’ checkpoint_best_loss.pth.tar} RENAMED
File without changes
checkpoints/ctaic/s1_det_instance/stage2/1/PLACEHOLDER DELETED
@@ -1,2 +0,0 @@
1
- PLACEHOLDER: replace this file with the real checkpoint (.pth.tar).
2
- See checkpoints/README.md for the expected filename and training copy commands.
 
 
 
checkpoints/ctaic/{s2_sem_panoptic/stage1/1/checkpoint_0.1313.pth.tar β†’ s1_det_instance/stage2/1/checkpoint_best_loss.pth.tar} RENAMED
File without changes
checkpoints/ctaic/s1_det_instance/stage2/2/PLACEHOLDER DELETED
@@ -1,2 +0,0 @@
1
- PLACEHOLDER: replace this file with the real checkpoint (.pth.tar).
2
- See checkpoints/README.md for the expected filename and training copy commands.
 
 
 
checkpoints/ctaic/{s2_sem_panoptic/stage1/2/checkpoint_0.1334.pth.tar β†’ s1_det_instance/stage2/2/checkpoint_best_loss.pth.tar} RENAMED
File without changes
checkpoints/ctaic/s1_det_instance/stage2/3/PLACEHOLDER DELETED
@@ -1,2 +0,0 @@
1
- PLACEHOLDER: replace this file with the real checkpoint (.pth.tar).
2
- See checkpoints/README.md for the expected filename and training copy commands.
 
 
 
checkpoints/ctaic/{s2_sem_panoptic/stage1/3/checkpoint_0.1564.pth.tar β†’ s1_det_instance/stage2/3/checkpoint_best_loss.pth.tar} RENAMED
File without changes
checkpoints/ctaic/s1_det_instance/stage2/4/PLACEHOLDER DELETED
@@ -1,2 +0,0 @@
1
- PLACEHOLDER: replace this file with the real checkpoint (.pth.tar).
2
- See checkpoints/README.md for the expected filename and training copy commands.
 
 
 
checkpoints/ctaic/{s2_sem_panoptic/stage1/4/checkpoint_0.1245.pth.tar β†’ s1_det_instance/stage2/4/checkpoint_best_loss.pth.tar} RENAMED
File without changes
checkpoints/ctaic/s2_sem_panoptic/stage1/1/PLACEHOLDER DELETED
@@ -1,2 +0,0 @@
1
- PLACEHOLDER: replace this file with the real checkpoint (.pth.tar).
2
- See checkpoints/README.md for the expected filename and training copy commands.
 
 
 
checkpoints/ctaic/{s3_det_pose/stage1/1/checkpoint_0.1313.pth.tar β†’ s2_sem_panoptic/stage1/1/checkpoint_best_loss.pth.tar} RENAMED
File without changes
checkpoints/ctaic/s2_sem_panoptic/stage1/2/PLACEHOLDER DELETED
@@ -1,2 +0,0 @@
1
- PLACEHOLDER: replace this file with the real checkpoint (.pth.tar).
2
- See checkpoints/README.md for the expected filename and training copy commands.
 
 
 
checkpoints/ctaic/{s3_det_pose/stage1/2/checkpoint_0.1334.pth.tar β†’ s2_sem_panoptic/stage1/2/checkpoint_best_loss.pth.tar} RENAMED
File without changes
checkpoints/ctaic/s2_sem_panoptic/stage1/3/PLACEHOLDER DELETED
@@ -1,2 +0,0 @@
1
- PLACEHOLDER: replace this file with the real checkpoint (.pth.tar).
2
- See checkpoints/README.md for the expected filename and training copy commands.
 
 
 
checkpoints/ctaic/{s3_det_pose/stage1/3/checkpoint_0.1564.pth.tar β†’ s2_sem_panoptic/stage1/3/checkpoint_best_loss.pth.tar} RENAMED
File without changes
checkpoints/ctaic/s2_sem_panoptic/stage1/4/PLACEHOLDER DELETED
@@ -1,2 +0,0 @@
1
- PLACEHOLDER: replace this file with the real checkpoint (.pth.tar).
2
- See checkpoints/README.md for the expected filename and training copy commands.
 
 
 
checkpoints/ctaic/{s3_det_pose/stage1/4/checkpoint_0.1245.pth.tar β†’ s2_sem_panoptic/stage1/4/checkpoint_best_loss.pth.tar} RENAMED
File without changes
checkpoints/ctaic/s2_sem_panoptic/stage2/1/PLACEHOLDER DELETED
@@ -1,2 +0,0 @@
1
- PLACEHOLDER: replace this file with the real checkpoint (.pth.tar).
2
- See checkpoints/README.md for the expected filename and training copy commands.
 
 
 
checkpoints/{taic/detection/1/checkpoint_0.1313.pth.tar β†’ ctaic/s2_sem_panoptic/stage2/1/checkpoint_best_loss.pth.tar} RENAMED
File without changes
checkpoints/ctaic/s2_sem_panoptic/stage2/2/PLACEHOLDER DELETED
@@ -1,2 +0,0 @@
1
- PLACEHOLDER: replace this file with the real checkpoint (.pth.tar).
2
- See checkpoints/README.md for the expected filename and training copy commands.
 
 
 
checkpoints/{taic/detection/2/checkpoint_0.1334.pth.tar β†’ ctaic/s2_sem_panoptic/stage2/2/checkpoint_best_loss.pth.tar} RENAMED
File without changes
checkpoints/ctaic/s2_sem_panoptic/stage2/3/PLACEHOLDER DELETED
@@ -1,2 +0,0 @@
1
- PLACEHOLDER: replace this file with the real checkpoint (.pth.tar).
2
- See checkpoints/README.md for the expected filename and training copy commands.
 
 
 
checkpoints/{taic/detection/3/checkpoint_0.1564.pth.tar β†’ ctaic/s2_sem_panoptic/stage2/3/checkpoint_best_loss.pth.tar} RENAMED
File without changes
checkpoints/ctaic/s2_sem_panoptic/stage2/4/PLACEHOLDER DELETED
@@ -1,2 +0,0 @@
1
- PLACEHOLDER: replace this file with the real checkpoint (.pth.tar).
2
- See checkpoints/README.md for the expected filename and training copy commands.
 
 
 
checkpoints/{taic/detection/4/checkpoint_0.1245.pth.tar β†’ ctaic/s2_sem_panoptic/stage2/4/checkpoint_best_loss.pth.tar} RENAMED
File without changes
checkpoints/ctaic/s3_det_pose/stage1/1/PLACEHOLDER DELETED
@@ -1,2 +0,0 @@
1
- PLACEHOLDER: replace this file with the real checkpoint (.pth.tar).
2
- See checkpoints/README.md for the expected filename and training copy commands.
 
 
 
checkpoints/{taic/instance/1/checkpoint_0.1313.pth.tar β†’ ctaic/s3_det_pose/stage1/1/checkpoint_best_loss.pth.tar} RENAMED
File without changes
checkpoints/ctaic/s3_det_pose/stage1/2/PLACEHOLDER DELETED
@@ -1,2 +0,0 @@
1
- PLACEHOLDER: replace this file with the real checkpoint (.pth.tar).
2
- See checkpoints/README.md for the expected filename and training copy commands.
 
 
 
checkpoints/{taic/instance/2/checkpoint_0.1334.pth.tar β†’ ctaic/s3_det_pose/stage1/2/checkpoint_best_loss.pth.tar} RENAMED
File without changes
checkpoints/ctaic/s3_det_pose/stage1/3/PLACEHOLDER DELETED
@@ -1,2 +0,0 @@
1
- PLACEHOLDER: replace this file with the real checkpoint (.pth.tar).
2
- See checkpoints/README.md for the expected filename and training copy commands.
 
 
 
checkpoints/{taic/instance/3/checkpoint_0.1564.pth.tar β†’ ctaic/s3_det_pose/stage1/3/checkpoint_best_loss.pth.tar} RENAMED
File without changes
checkpoints/ctaic/s3_det_pose/stage1/4/PLACEHOLDER DELETED
@@ -1,2 +0,0 @@
1
- PLACEHOLDER: replace this file with the real checkpoint (.pth.tar).
2
- See checkpoints/README.md for the expected filename and training copy commands.
 
 
 
checkpoints/{taic/instance/4/checkpoint_0.1245.pth.tar β†’ ctaic/s3_det_pose/stage1/4/checkpoint_best_loss.pth.tar} RENAMED
File without changes
checkpoints/ctaic/s3_det_pose/stage2/1/PLACEHOLDER DELETED
@@ -1,2 +0,0 @@
1
- PLACEHOLDER: replace this file with the real checkpoint (.pth.tar).
2
- See checkpoints/README.md for the expected filename and training copy commands.
 
 
 
checkpoints/ctaic/s3_det_pose/stage2/1/checkpoint_best_loss.pth.tar ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a8c869589438d291a34911e8dffe8a462e67d0587f94773cee66eb9ecd0a80b1
3
+ size 70789902