aceOfDiamonds commited on
Commit
a5764d2
·
1 Parent(s): 402e59e

Add the model card standard

Browse files

Every model repository's README follows one shape: four sections generated
from the tree and the config.json files, plus hand-written free sections for
what the artifacts cannot express. Records which facts a card must not
restate, so a card and the files it describes cannot disagree.

Files changed (2) hide show
  1. MODEL_CARD.md +167 -0
  2. README.md +4 -3
MODEL_CARD.md ADDED
@@ -0,0 +1,167 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Model card standard
2
+
3
+ Every `software-mansion/react-native-executorch-*` repository carries a
4
+ `README.md` in this shape. Four sections are **generated** from the repository
5
+ tree and its `config.json` files, so they cannot drift from what is published.
6
+ Anything a model needs beyond them is a **free section**, written by hand and
7
+ kept as-is.
8
+
9
+ Before this standard the estate had 12 distinct heading sequences, four
10
+ heading-level conventions, and five different ExecuTorch versions claimed in
11
+ prose while 28 of 54 cards claimed none.
12
+
13
+ ## Front matter
14
+
15
+ Hugging Face card metadata. Required on every repository:
16
+
17
+ | key | value |
18
+ | --- | --- |
19
+ | `license` | An SPDX-style Hub identifier for the **upstream** license, not ours. Use `other` plus `license_name` and `license_link` when the upstream license has no Hub identifier. |
20
+ | `pipeline_tag` | The Hub task this model serves, so it appears in task search. |
21
+ | `library_name` | Always `executorch`. |
22
+
23
+ Optional: `base_model`, `language`, `tags`. Keep the same identifier here and in
24
+ the `license` field of every `config.json` in the repository; they are checked
25
+ against each other.
26
+
27
+ ## Generated sections
28
+
29
+ ### 1. Title and introduction
30
+
31
+ The `# ` title is the repository slug with the `react-native-executorch-`
32
+ prefix dropped. One paragraph naming the library, then the upstream model on
33
+ its own line:
34
+
35
+ ```markdown
36
+ # <slug>
37
+
38
+ This repository hosts the **<slug>** models exported for the
39
+ [React Native ExecuTorch](https://www.npmjs.com/package/react-native-executorch)
40
+ library as ExecuTorch `.pte` programs, ready to run on device.
41
+
42
+ Upstream model: [<name>](<url>)
43
+ ```
44
+
45
+ Omit the upstream line when there is no single upstream to point at, rather
46
+ than guessing a URL.
47
+
48
+ ### 2. Variants
49
+
50
+ One row per published `.pte`, read from the `variants` array of every
51
+ `config.json`.
52
+
53
+ ```markdown
54
+ ## Variants
55
+
56
+ | Path | Backend | Precision |
57
+ | --- | --- | --- |
58
+ | `xnnpack/model_xnnpack_fp32.pte` | xnnpack | fp32 |
59
+ | `xnnpack/model_xnnpack_int8.pte` | xnnpack | int8 |
60
+ | `coreml/model_coreml_fp16.pte` | coreml | fp16 |
61
+ ```
62
+
63
+ A variant that ships several `.pte` files declares them as a `components` map
64
+ of role to filename, and contributes one row per component. Those repositories
65
+ gain a `Component` column naming the role; repositories without components do
66
+ not carry the column, so the common case stays three columns wide.
67
+
68
+ ```markdown
69
+ | Path | Component | Backend | Precision |
70
+ | --- | --- | --- | --- |
71
+ | `coreml/clip_image_coreml_fp16.pte` | image | coreml | fp16 |
72
+ | `coreml/clip_text_coreml_fp16.pte` | text | coreml | fp16 |
73
+ ```
74
+
75
+ The precision token is the whole story: `precisions.json` is authoritative for
76
+ which tokens are quantized, so the card does not repeat that partition. Nor
77
+ does it name a default, because the library resolves one per platform and a
78
+ second answer here would only compete with it.
79
+
80
+ ### 3. Repository structure
81
+
82
+ Every tracked file except `README.md` and dotfiles, with its size, so a reader
83
+ can see the download cost before fetching.
84
+
85
+ ```markdown
86
+ ## Repository structure
87
+
88
+ ​```
89
+ coreml/config.json 1 kB
90
+ coreml/model_coreml_fp16.pte 8.3 MB
91
+ xnnpack/config.json 3 kB
92
+ xnnpack/model_xnnpack_fp32.pte 31.1 MB
93
+ ​```
94
+ ```
95
+
96
+ Sizes render as `B`, `kB`, `MB` or `GB`.
97
+
98
+ ### 4. Compatibility
99
+
100
+ Fixed text, parameterised only by the ExecuTorch runtime version. It states
101
+ the runtime the files are **published for**, and deliberately does not claim
102
+ the version each file was exported with: that is unrecorded for older exports,
103
+ and the five conflicting numbers the cards used to carry were all stale.
104
+
105
+ No react-native-executorch version appears here. Naming one would force an
106
+ edit to every card on each library release even when neither the runtime nor
107
+ the files changed, which is exactly the drift this standard exists to stop.
108
+
109
+ ```markdown
110
+ ## Compatibility
111
+
112
+ These files are published for the **ExecuTorch v1.4.1** runtime. ExecuTorch
113
+ gives no forward compatibility guarantee, so an older runtime may fail to load
114
+ them.
115
+
116
+ To use them in React Native ExecuTorch, pass the model constant shipped in the
117
+ library's model registry to the corresponding task pipeline. See the
118
+ [documentation](https://docs.swmansion.com/react-native-executorch/docs/fundamentals/loading-models).
119
+
120
+ To load these files in your own ExecuTorch runtime, read the
121
+ [compatibility note](https://github.com/pytorch/executorch/blob/main/runtime/COMPATIBILITY.md)
122
+ first.
123
+ ```
124
+
125
+ Bump the version here only when the runtime the files are published for
126
+ changes.
127
+
128
+ ## Free sections
129
+
130
+ Everything a model needs that neither the skeleton nor the repository's JSON
131
+ can express, appended after Compatibility at `##` level and left exactly as
132
+ written.
133
+
134
+ - `Export notes`, `CoreML notes (iOS)`: export decisions and per-backend
135
+ caveats. These live nowhere else, so the card is their home.
136
+ - `Model details`, `Languages`, `Classes (N)`: **only** when the model is
137
+ niche or the upstream link does not already cover it. For a model with a
138
+ well-documented upstream this duplicates a page we do not control and
139
+ cannot keep in sync.
140
+
141
+ Do not write a free section for:
142
+
143
+ - **The I/O contract.** The `config.json` files declare it, so restating it
144
+ here breaks rule 5. (Two shapes are in play today, the schema and the older
145
+ one reporting only upper bounds for dims, which is a reason to converge
146
+ them, not to re-document I/O by hand in 54 places.)
147
+ - **Latency or benchmark numbers.** They belong on the docs page, where they
148
+ are versioned and compared across models. Nobody looks in a model card for
149
+ them, and a stale number here is worse than none.
150
+ - **Required preprocessing.** `models.ts` declares it, and these files are
151
+ published to be consumed through the library, so its readers already have
152
+ the authoritative copy. A second copy here is one more thing to keep in
153
+ sync for an audience that does not need it.
154
+
155
+ Write a free section only when the answer to "how do I call this correctly?"
156
+ follows from neither the variant table nor the JSON.
157
+
158
+ ## Rules
159
+
160
+ 1. Heading levels are fixed: one `#` title, everything else `##`.
161
+ 2. Never hand-edit a generated section. Fix the tree or the `config.json` and
162
+ re-render, so the card and the artifacts cannot disagree.
163
+ 3. One ExecuTorch version appears in a card, in Compatibility.
164
+ 4. Free sections carry the model-specific truth. Boilerplate belongs in the
165
+ generated skeleton, where it is written once.
166
+ 5. Nothing in a card restates a fact the artifacts already hold. Derive it
167
+ instead, so the two cannot disagree.
README.md CHANGED
@@ -18,13 +18,14 @@ model repositories under
18
  | -------------------- | ----------------------------------------------------------------------------------------- |
19
  | `config.schema.json` | JSON Schema for the per-backend `config.json` files shipped alongside each `.pte`. |
20
  | `precisions.json` | Authoritative quantized / non-quantized partition for precision tokens used in file names. |
 
21
  | `README.md` | This file. |
22
 
23
  ## Source of truth
24
 
25
- The artifacts in this repository are the spec. Edit `config.schema.json` and
26
- `precisions.json` here, and treat them as authoritative for every published
27
- `config.json`.
28
 
29
  ## Versioning
30
 
 
18
  | -------------------- | ----------------------------------------------------------------------------------------- |
19
  | `config.schema.json` | JSON Schema for the per-backend `config.json` files shipped alongside each `.pte`. |
20
  | `precisions.json` | Authoritative quantized / non-quantized partition for precision tokens used in file names. |
21
+ | `MODEL_CARD.md` | The README standard every model repository's card follows. |
22
  | `README.md` | This file. |
23
 
24
  ## Source of truth
25
 
26
+ The artifacts in this repository are the spec. Edit `config.schema.json`,
27
+ `precisions.json` and `MODEL_CARD.md` here, and treat them as authoritative
28
+ for every published `config.json` and `README.md`.
29
 
30
  ## Versioning
31