| --- |
| license: apache-2.0 |
| --- |
| # Model Card for TurboSparse-Mistral |
| The [TurboSparse-Mistral](https://arxiv.org/abs/2406.05955) Large Language Model (LLM) is an sparsified version of the Mixtral. |
|
|
| <img src="takeaway.png" alt="avatar" width="300" height="200"/> |
|
|
| The average performance is evaluated using benchmarks from the OpenLLM Leaderboard. |
|
|
| ## Inference |
|
|
| Our code for accelerating TurboSparse-Mixtral is currently being refined. Stay tuned! Now you can run this model like dense model. |
|
|
| ## Chat-Template |
|
|
| During sparsification, we also utilize some SFT datasets. |
| We take ChatML as our chat template: |
| ``` |
| <|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n |
| ``` |
|
|
| ## Allow Finetuning |
|
|
| As we merged the predictors for FFN neurons in models, you can finetune TurboSparse-Mistral with any framework and algorithm. |
|
|
| ## Limitations |
| * TurboSparse, having just undergone training with 150B tokens, may still exhibit performance gaps in certain tasks. |
| * The TurboSparse model has only been trained on English-language datasets, hence its capabilities in other languages are still lacking. |
| * The model may produce unexpected outputs due to its small size, limited training tokens and probabilistic generation paradigm. |
|
|
| ## License |
|
|
| The model is licensed under Apache-2.0, while model weights are fully open for academic research and also allow **free** commercial usage. |
