Instructions to use HuggingFaceM4/idefics-9b-instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use HuggingFaceM4/idefics-9b-instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="HuggingFaceM4/idefics-9b-instruct")

# Load model directly
from transformers import AutoProcessor, AutoModelForImageTextToText

processor = AutoProcessor.from_pretrained("HuggingFaceM4/idefics-9b-instruct")
model = AutoModelForImageTextToText.from_pretrained("HuggingFaceM4/idefics-9b-instruct")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use HuggingFaceM4/idefics-9b-instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "HuggingFaceM4/idefics-9b-instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HuggingFaceM4/idefics-9b-instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/HuggingFaceM4/idefics-9b-instruct

SGLang

How to use HuggingFaceM4/idefics-9b-instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "HuggingFaceM4/idefics-9b-instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HuggingFaceM4/idefics-9b-instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "HuggingFaceM4/idefics-9b-instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HuggingFaceM4/idefics-9b-instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use HuggingFaceM4/idefics-9b-instruct with Docker Model Runner:
```
docker model run hf.co/HuggingFaceM4/idefics-9b-instruct
```

VictorSanh commited on Aug 19, 2023

Commit

d9566ea

1 Parent(s): 7a17dc0

Update readme and doc from the 80b repo

Browse files

Files changed (6) hide show

.gitattributes +2 -0
README.md +8 -4
assets/Figure_Evals_IDEFICS.png +3 -0
assets/IDEFICS.png +3 -0
assets/Idefics_colab.png +3 -0
assets/guarding_baguettes.png +3 -0

.gitattributes CHANGED Viewed

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text

 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/Figure_Evals_IDEFICS.png filter=lfs diff=lfs merge=lfs -text
+*.png filter=lfs diff=lfs merge=lfs -text

README.md CHANGED Viewed

@@ -19,6 +19,8 @@ pipeline_tag: text-generation
 # IDEFICS
 IDEFICS (**I**mage-aware **D**ecoder **E**nhanced à la **F**lamingo with **I**nterleaved **C**ross-attention**S**) is an open-access reproduction of [Flamingo](https://huggingface.co/papers/2204.14198), a closed-source visual language model developed by Deepmind. Like GPT-4, the multimodal model accepts arbitrary sequences of image and text inputs and produces text outputs. IDEFICS is built solely on public available data and models.
 The model can answer questions about images, describe visual contents, create stories grounded on multiple images, or simply behave as a pure language model without visual inputs.
@@ -29,7 +31,7 @@ We also fine-tune these base models on a mixture of supervised and instruction f
 Read more about some of the technical challenges we encountered during training IDEFICS [here](https://github.com/huggingface/m4-logs/blob/master/memos/README.md).
-*How do I pronounce the name? [Youtube tutorial](https://www.youtube.com/watch?v=YKO0rWnPN2I&ab_channel=FrenchPronunciationGuide)*
 # Model Details
@@ -39,7 +41,7 @@ Read more about some of the technical challenges we encountered during training
 - **License:** see [License section](#license)
 - **Parent Model:** [laion/CLIP-ViT-H-14-laion2B-s32B-b79K](https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K) and [huggyllama/llama-65b](https://huggingface.co/huggyllama/llama-65b)
 - **Resources for more information:**
-    - [GitHub Repo](https://github.com/huggingface/m4/)
     - Description of [OBELICS](https://huggingface.co/datasets/HuggingFaceM4/OBELICS): [OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents
 ](https://huggingface.co/papers/2306.16527)
     - Original Paper: [Flamingo: a Visual Language Model for Few-Shot Learning](https://huggingface.co/papers/2204.14198)
@@ -420,9 +422,11 @@ We release the additional weights we trained under an MIT license.
 }
 ```
-# Model Card Authors
-Stas Bekman, Léo Tronchon, Hugo Laurençon, Lucile Saulnier, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Giada Pistilli, Yacine Jernite, Sasha Luccioni, Ezi Ozoani, Younes Belkada, Sylvain Gugger, Amy E. Roberts, Lysandre Debut, Arthur Zucker, Lewis Tunstall, Zach Mueller, Sourab Mangrulkar, Yuvraj Sharma, Dawood Khan, Abubakar Abid, Daniel Van Strien, Chunte Lee, Douwe Kiela, Alexander M. Rush, Matthieu Cord, Victor Sanh
 # Model Card Contact

 # IDEFICS
+*How do I pronounce the name? [Youtube tutorial](https://www.youtube.com/watch?v=YKO0rWnPN2I&ab_channel=FrenchPronunciationGuide)*
 IDEFICS (**I**mage-aware **D**ecoder **E**nhanced à la **F**lamingo with **I**nterleaved **C**ross-attention**S**) is an open-access reproduction of [Flamingo](https://huggingface.co/papers/2204.14198), a closed-source visual language model developed by Deepmind. Like GPT-4, the multimodal model accepts arbitrary sequences of image and text inputs and produces text outputs. IDEFICS is built solely on public available data and models.
 The model can answer questions about images, describe visual contents, create stories grounded on multiple images, or simply behave as a pure language model without visual inputs.
 Read more about some of the technical challenges we encountered during training IDEFICS [here](https://github.com/huggingface/m4-logs/blob/master/memos/README.md).
+**Try out the [demo](https://huggingface.co/spaces/HuggingFaceM4/idefics_playground)!**
 # Model Details
 - **License:** see [License section](#license)
 - **Parent Model:** [laion/CLIP-ViT-H-14-laion2B-s32B-b79K](https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K) and [huggyllama/llama-65b](https://huggingface.co/huggyllama/llama-65b)
 - **Resources for more information:**
+    <!-- - [GitHub Repo](https://github.com/huggingface/m4/) -->
     - Description of [OBELICS](https://huggingface.co/datasets/HuggingFaceM4/OBELICS): [OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents
 ](https://huggingface.co/papers/2306.16527)
     - Original Paper: [Flamingo: a Visual Language Model for Few-Shot Learning](https://huggingface.co/papers/2204.14198)
 }
 ```
+# Model Builders, Card Authors, and contributors
+The core team (*) was supported in many different ways by these contributors at Hugging Face:
+Stas Bekman*, Léo Tronchon*, Hugo Laurençon*, Lucile Saulnier*, Amanpreet Singh*, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Daniel Van Strien, Giada Pistilli, Yacine Jernite, Sasha Luccioni, Ezi Ozoani, Younes Belkada, Sylvain Gugger, Amy E. Roberts, Lysandre Debut, Arthur Zucker,  Nicolas Patry, Lewis Tunstall, Zach Mueller, Sourab Mangrulkar, Chunte Lee, Yuvraj Sharma, Dawood Khan, Abubakar Abid, Ali Abid, Freddy Boulton, Omar Sanseviero, Carlos Muñoz Ferrandis, Guillaume Salou, Guillaume Legendre, Quentin Lhoest, Douwe Kiela, Alexander M. Rush, Matthieu Cord, Julien Chaumond, Thomas Wolf, Victor Sanh*
 # Model Card Contact

assets/Figure_Evals_IDEFICS.png ADDED Viewed

Git LFS Details

SHA256: 5adef77aef167bd4ae649d18b989f7b435dc1ad8cbe5679a9a0c14bee4a8d67e
Pointer size: 132 Bytes
Size of remote file: 2.04 MB

assets/IDEFICS.png ADDED Viewed

Git LFS Details

SHA256: f19cb840364022ca231f9bbd43f8ba05bda417134a0586c28853ab29a5fa55e6
Pointer size: 131 Bytes
Size of remote file: 209 kB

assets/Idefics_colab.png ADDED Viewed

Git LFS Details

SHA256: 0c8edfcda54aff7d9d6a462521da0686401ee34442c533aa782a3d15a6c3945a
Pointer size: 132 Bytes
Size of remote file: 1.37 MB

assets/guarding_baguettes.png ADDED Viewed

Git LFS Details

SHA256: d88ae51fd8cacd3950985df8e01566c40f011ba7d2d2bb9108f54dce1edf9672
Pointer size: 131 Bytes
Size of remote file: 455 kB