Improve the WorldDiT model card layout

#2
by bagellabs - opened
Files changed (1) hide show
  1. README.md +31 -85
README.md CHANGED
@@ -12,11 +12,9 @@ tags:
12
  - robotic-manipulation
13
  - imitation-learning
14
  - diffusion-transformer
 
15
  - flow-matching
16
- - pareto-frontier
17
- - parameter-efficiency
18
  inference: false
19
- license: cc-by-4.0
20
  widget:
21
  - example_title: "LIBERO Spatial, task 5"
22
  text: "Successful rollout, front view."
@@ -36,26 +34,23 @@ widget:
36
  url: "https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/worlddit/model-card/v1/worlddit_libero_10_frontview_task06_episode01.mp4"
37
  ---
38
 
39
- Bagel Labs is a physical AI research lab based in San Francisco and Toronto. It has developed an architecture that uses dramatically less data and compute to train while delivering a state-of-the-art model-size/performance tradeoff. WorldDiT demonstrates the model advantage; Paris 1.0 and Paris 2.0 demonstrate the training advantage.
40
-
41
  <p align="center">
42
  <img src="https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/paris2/model-card/v1/bagel_labs_logo.png" alt="Bagel Labs">
43
  </p>
44
 
45
- <h1 align="center">WorldDiT: A Unified Diffusion Backbone for<br>World and Action Modeling</h1>
 
 
46
 
47
  <p align="center">
48
- <a href="https://arxiv.org/abs/2607.23909">
49
- <img src="https://img.shields.io/badge/arXiv-2607.23909-B31B1B.svg?logo=arxiv&logoColor=white" alt="WorldDiT paper on arXiv">
50
- </a>
51
  </p>
52
 
53
  <p align="center">
54
- WorldDiT couples continuous action generation with auxiliary future normalized
55
- RGB patch prediction in one diffusion transformer. The architecture is designed
56
- as a general backbone for world and action modeling, while the current release
57
- evaluates it on LIBERO and provides four checkpoints, a self contained
58
- inference runtime, and an evaluator.
59
  </p>
60
 
61
  ## See WorldDiT act
@@ -103,8 +98,8 @@ Each clip covers a different LIBERO suite and camera view.
103
 
104
  | Reported LIBERO result | Released model |
105
  |---|---|
106
- | **94.9 percent** mean success<br>**1,898 of 2,000** successful episodes | **399.084 million** total parameters<br>**135.107 million** trainable parameters |
107
- | **98.0 percent** Spatial<br>**97.0 percent** Object | **Three** observation steps<br>**Seven** predicted actions |
108
  | **92.8 percent** Goal<br>**91.8 percent** Long | **Three** actions executed before replanning<br>**Seven** action dimensions |
109
 
110
  | Checkpoints | Runtime | Encoders and environment |
@@ -115,23 +110,9 @@ The repository is self contained for WorldDiT inference. LIBERO provides the
115
  benchmark environments, assets, task definitions, and initial states.
116
 
117
  The released runtime and checkpoints were revalidated from a clean installation
118
- on eight RTX Pro 6000 Blackwell GPUs. The reported aggregate covers five
119
- hundred simulator episodes per suite. Three hundred episodes per suite informed
120
- staged checkpoint selection, while two hundred episodes per suite were disjoint
121
- from selection.
122
-
123
- ### Parameter count and reported success
124
-
125
- Among methods with complete four suite averages, WorldDiT lies on the reported
126
- Pareto frontier for total model parameters and mean LIBERO success.
127
-
128
- <p align="center">
129
- <img src="https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/worlddit/model-card/v2/worlddit_parameter_efficiency.png" alt="Reported mean LIBERO success against total instantiated model parameters">
130
- </p>
131
-
132
- <p align="center">
133
- <sub>Reported LIBERO success against total model parameters for 24 methods. The line connects methods on the Pareto frontier with complete four suite averages. Because the methods follow different published evaluation protocols, the figure summarizes published results rather than a direct comparison under one common evaluation protocol.</sub>
134
- </p>
135
 
136
  ## Run a smoke test
137
 
@@ -168,34 +149,24 @@ python eval.py \
168
  ```
169
 
170
  A successful smoke test confirms that the environment, checkpoint, visual
171
- encoders, simulator, and rendering path load together. Full benchmark reporting
172
- uses complete suite evaluations.
173
 
174
  ## How WorldDiT works
175
 
176
- Each of three recent observation steps contributes primary and wrist images
177
- together with robot state, while one language instruction conditions the
178
- sequence. During training, one diffusion transformer backbone learns a seven
179
- step action chunk together with an auxiliary future normalized RGB patch
180
- target. At deployment, the encoded history conditions the action velocity field
181
- directly. RGB patch token construction and RGB prediction head evaluation
182
- remain outside the inference graph, concentrating computation on action
183
- generation. The controller executes the first three predicted actions, observes
184
- again, and replans.
185
 
186
- <p align="center">
187
- <img src="https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/worlddit/model-card/v2/worlddit_inference_pipeline.png?v=09d6508" alt="WorldDiT inference pipeline">
188
- </p>
189
-
190
- <p align="center">
191
- <sub>WorldDiT inference pipeline. The encoded observation history conditions action generation through twenty flow steps. The controller executes the first three actions from each seven action chunk, then updates the window and replans.</sub>
192
- </p>
193
 
194
  | Training | Deployment |
195
  |---|---|
196
- | Action and future normalized RGB patch targets are learned by one backbone | Encoded history conditions the action velocity field |
197
  | Seven action steps are supervised | Seven actions are predicted |
198
- | Future normalized RGB patch supervision is present | RGB patch tokens and the RGB prediction head remain outside the inference graph |
199
  | The complete training objective is active | Three actions execute before replanning |
200
 
201
  ## Reference
@@ -223,8 +194,8 @@ CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python eval.py \
223
 
224
  Each GPU receives an independent progress bar. After all workers finish, rank 0
225
  prints per task and overall success rates and writes a structured
226
- `results.json`. Use a new output directory for each evaluation to preserve
227
- earlier results.
228
 
229
  Supported suites.
230
 
@@ -257,8 +228,7 @@ libero_10
257
  ```
258
 
259
  `dependencies/` contains the frozen visual and language encoder weights needed
260
- by the released WorldDiT runtime. The repository contains every model weight
261
- required for inference.
262
 
263
  </details>
264
 
@@ -289,8 +259,8 @@ Evaluation uses the final temporal slot of the predicted action tensor.
289
 
290
  | Component | Specification |
291
  |---|---|
292
- | Backbone | WorldDiT diffusion transformer |
293
- | Observation context | 3 observation steps |
294
  | Action horizon | 7 actions |
295
  | Action dimension | 7 |
296
  | Action aggregation | Temporal ensembling |
@@ -305,32 +275,8 @@ Evaluation uses the final temporal slot of the predicted action tensor.
305
 
306
  | Intended use | Scope of the release |
307
  |---|---|
308
- | Research on world and action modeling for language conditioned robot manipulation. The architecture supports continuous action generation with auxiliary future normalized RGB patch prediction. | The current release evaluates WorldDiT in LIBERO simulation under the released protocol and provides checkpoints for all four suites. |
309
- | Architecture research, reproduction, and evaluation of multimodal diffusion backbones for robot manipulation. | Real robot reliability, safety, and transfer across embodiments require dedicated future evaluation. The present experiments evaluate the integrated WorldDiT system. Targeted ablations are required to attribute performance to the future normalized RGB patch objective. Total instantiated parameter count characterizes model scale. Training cost, deployment latency, and runtime efficiency require dedicated measurements. |
310
-
311
- ## Citation
312
-
313
- If you use WorldDiT in your research, please cite the paper.
314
-
315
- ```bibtex
316
- @article{260723909,
317
- title={{WorldDiT: A Unified Diffusion Architecture for World and Action Modeling}},
318
- author={Sen Wang and R. Gnana Praveen and Bidhan Roy and Marcos Villagra},
319
- year={{2026}},
320
- eprint={{2607.23909}},
321
- archivePrefix={{arXiv}}
322
- }
323
- ```
324
-
325
- ## License
326
-
327
- The WorldDiT checkpoints, model card, and original release materials are
328
- licensed under [Creative Commons Attribution 4.0
329
- International](https://creativecommons.org/licenses/by/4.0/). You may copy,
330
- redistribute, and adapt them, including commercially, with appropriate credit
331
- to Bagel Labs and the WorldDiT authors, a link to the license, and an indication
332
- of any changes. Third party dependencies and assets remain governed by their
333
- upstream licenses.
334
 
335
  ## Authors and contact
336
 
 
12
  - robotic-manipulation
13
  - imitation-learning
14
  - diffusion-transformer
15
+ - diffusion-policy
16
  - flow-matching
 
 
17
  inference: false
 
18
  widget:
19
  - example_title: "LIBERO Spatial, task 5"
20
  text: "Successful rollout, front view."
 
34
  url: "https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/worlddit/model-card/v1/worlddit_libero_10_frontview_task06_episode01.mp4"
35
  ---
36
 
 
 
37
  <p align="center">
38
  <img src="https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/paris2/model-card/v1/bagel_labs_logo.png" alt="Bagel Labs">
39
  </p>
40
 
41
+ <h1 align="center">WorldDiT</h1>
42
+
43
+ <h2 align="center">One diffusion backbone learns what to do and what comes next.</h2>
44
 
45
  <p align="center">
46
+ WorldDiT learns continuous robot action chunks and a future visual target
47
+ through one shared diffusion transformer. Deployment keeps only the action
48
+ path.
49
  </p>
50
 
51
  <p align="center">
52
+ This release includes four LIBERO checkpoints, a self contained inference
53
+ runtime, and an evaluator for reproducing the reported suite results.
 
 
 
54
  </p>
55
 
56
  ## See WorldDiT act
 
98
 
99
  | Reported LIBERO result | Released model |
100
  |---|---|
101
+ | **94.9 percent** selection aware mean<br>**1,898 of 2,000** successful episodes | **399.084 million** total parameters<br>**135.107 million** trainable parameters |
102
+ | **98.0 percent** Spatial<br>**97.0 percent** Object | **Three** observation frames<br>**Seven** predicted actions |
103
  | **92.8 percent** Goal<br>**91.8 percent** Long | **Three** actions executed before replanning<br>**Seven** action dimensions |
104
 
105
  | Checkpoints | Runtime | Encoders and environment |
 
110
  benchmark environments, assets, task definitions, and initial states.
111
 
112
  The released runtime and checkpoints were revalidated from a clean installation
113
+ on eight RTX Pro 6000 Blackwell GPUs. The result is selection aware because
114
+ three hundred episodes per suite informed staged checkpoint selection before
115
+ the final five hundred episode score was assembled.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
116
 
117
  ## Run a smoke test
118
 
 
149
  ```
150
 
151
  A successful smoke test confirms that the environment, checkpoint, visual
152
+ encoders, simulator, and rendering path load together. It is not a benchmark
153
+ result.
154
 
155
  ## How WorldDiT works
156
 
157
+ WorldDiT uses three recent observations, robot state, and language as context.
158
+ During training, one diffusion transformer learns a seven step action chunk and
159
+ an auxiliary future visual target. During deployment, the future visual path is
160
+ absent. The policy executes the first three predicted actions, observes again,
161
+ and replans.
 
 
 
 
162
 
163
+ > Future visual prediction is a training signal, not a deployment path.
 
 
 
 
 
 
164
 
165
  | Training | Deployment |
166
  |---|---|
167
+ | Action and future visual targets share one backbone | Only the action path remains |
168
  | Seven action steps are supervised | Seven actions are predicted |
169
+ | Future visual supervision is present | No future visual output is requested |
170
  | The complete training objective is active | Three actions execute before replanning |
171
 
172
  ## Reference
 
194
 
195
  Each GPU receives an independent progress bar. After all workers finish, rank 0
196
  prints per task and overall success rates and writes a structured
197
+ `results.json`. Output directories must be new so an earlier evaluation is
198
+ never overwritten.
199
 
200
  Supported suites.
201
 
 
228
  ```
229
 
230
  `dependencies/` contains the frozen visual and language encoder weights needed
231
+ by the released policy. No additional model downloads are required.
 
232
 
233
  </details>
234
 
 
259
 
260
  | Component | Specification |
261
  |---|---|
262
+ | Policy | WorldDiT diffusion transformer |
263
+ | Observation context | 3 frames |
264
  | Action horizon | 7 actions |
265
  | Action dimension | 7 |
266
  | Action aggregation | Temporal ensembling |
 
275
 
276
  | Intended use | Scope of the release |
277
  |---|---|
278
+ | Research on language conditioned robot manipulation in the LIBERO simulator. The released checkpoints support reproduction, evaluation, and architecture research across the four released suites. | The results describe LIBERO simulation under the released evaluation protocol. They do not establish real robot reliability, safety, or transfer across embodiments. |
279
+ | The released checkpoints cover all four LIBERO suites. | The release does not isolate the causal contribution of the future visual target. Total parameter count does not measure training cost, deployment latency, or runtime efficiency. |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
280
 
281
  ## Authors and contact
282