Instructions to use codellama/CodeLlama-70b-Instruct-hf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use codellama/CodeLlama-70b-Instruct-hf with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="codellama/CodeLlama-70b-Instruct-hf")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("codellama/CodeLlama-70b-Instruct-hf")
model = AutoModelForCausalLM.from_pretrained("codellama/CodeLlama-70b-Instruct-hf")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use codellama/CodeLlama-70b-Instruct-hf with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "codellama/CodeLlama-70b-Instruct-hf"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "codellama/CodeLlama-70b-Instruct-hf",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/codellama/CodeLlama-70b-Instruct-hf

SGLang

How to use codellama/CodeLlama-70b-Instruct-hf with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "codellama/CodeLlama-70b-Instruct-hf" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "codellama/CodeLlama-70b-Instruct-hf",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "codellama/CodeLlama-70b-Instruct-hf" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "codellama/CodeLlama-70b-Instruct-hf",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use codellama/CodeLlama-70b-Instruct-hf with Docker Model Runner:
```
docker model run hf.co/codellama/CodeLlama-70b-Instruct-hf
```

Shouldn't tokenizer.eos be set to < step >?

#12

by RonanMcGovern - opened Jan 30, 2024

Discussion

RonanMcGovern

Jan 30, 2024

Or at least < step > has to be set as a stop token. Otherwise generation doesn't stop.

pcuenq

Code Llama org Jan 30, 2024

Works for me in HuggingChat :) Can you explain how you are using it?

RonanMcGovern

Jan 30, 2024

•

edited Jan 30, 2024

I think it's working in HuggingChat because a stop token of <step> has been set.

That's a fine approach too.

But more generically, if eos_token is set to <step> that is a little more robust.

Edit: replaced <step> as without the backticks it gets eaten by html/markdown. Thanks @unphased

unphased

Jan 30, 2024

•

edited Jan 30, 2024

@RonanMcGovern markdown accepts HTML so i think your comment's text had a "html tag" eaten up by markdown, in the middle of where you have written "a stop token of has been set.". I'll note the preferred way to stop this html interpretation is to wrap in markdown backtick inline code quotes.

Pretty curious how to get this thing to prompt properly. I'm also seeing some repeating output. Though I know I'm not prompting it properly yet. At least it does run though, with both llama.cpp on apple silicon and exl2 on dual 3090s. When I saw repeating in exl2 it was saying EOT: true Source: assistant before repeating itself.

RonanMcGovern

Jan 30, 2024

yeah, same, I was seeing EOT.

The two solutions I used were either to set a stop token OR, more robustly, just set eos to be <step>

pcuenq

Code Llama org Jan 31, 2024

Hi @unphased , @RonanMcGovern , thanks for sharing your experience! Would you mind posting a prompt that exhibits the behaviour you describe with the EOT token? I'd like to verify if this may be caused by differences in the prompt format :)

RonanMcGovern

Feb 1, 2024

To be clear, the EOT token appears after <step>, so if the eos or a stop token is set, then I don't see the EOT token.

Most prompts, e.g. "Write a piece of code to print the first 10 prime numbers of the fib series" with apply.chat_template will continue generating (and often that continuation contains EOT, after <step>).

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment