Instructions to use cortexso/smollm2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use cortexso/smollm2 with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="cortexso/smollm2",
	filename="smollm2-1.7b-instruct-q2_k.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use cortexso/smollm2 with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf cortexso/smollm2:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf cortexso/smollm2:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf cortexso/smollm2:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf cortexso/smollm2:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf cortexso/smollm2:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf cortexso/smollm2:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf cortexso/smollm2:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf cortexso/smollm2:Q4_K_M

Use Docker

docker model run hf.co/cortexso/smollm2:Q4_K_M

LM Studio
Jan

vLLM

How to use cortexso/smollm2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "cortexso/smollm2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cortexso/smollm2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/cortexso/smollm2:Q4_K_M

Ollama
How to use cortexso/smollm2 with Ollama:
```
ollama run hf.co/cortexso/smollm2:Q4_K_M
```

Unsloth Studio new

How to use cortexso/smollm2 with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for cortexso/smollm2 to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for cortexso/smollm2 to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for cortexso/smollm2 to start chatting

Docker Model Runner
How to use cortexso/smollm2 with Docker Model Runner:
```
docker model run hf.co/cortexso/smollm2:Q4_K_M
```

Lemonade

How to use cortexso/smollm2 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull cortexso/smollm2:Q4_K_M

Run and chat with the model

lemonade run user.smollm2-Q4_K_M

List all available models

lemonade list

Overview

SmolLM2 is a family of compact language models available in three sizes: 135M, 360M, and 1.7B parameters. These models are designed to solve a wide range of tasks while being lightweight enough for on-device deployment. More details can be found in the SmolLM2 paper.

The 1.7B variant demonstrates significant improvements over its predecessor, SmolLM1-1.7B, especially in instruction following, knowledge retention, reasoning, and mathematical problem-solving. It was trained on 11 trillion tokens using a diverse dataset combination, including FineWeb-Edu, DCLM, The Stack, and newly curated mathematics and coding datasets that will be released soon.

The instruct version of SmolLM2 was developed through supervised fine-tuning (SFT) using a mix of public datasets and curated proprietary datasets. It further benefits from Direct Preference Optimization (DPO) using UltraFeedback.

Additionally, the instruct model supports tasks such as text rewriting, summarization, and function calling, enabled by datasets from Argilla, including Synth-APIGen-v0.1. The SFT dataset is available at: SmolTalk SFT Dataset.

For further details, visit the SmolLM2 GitHub repository, where you will find resources for pre-training, post-training, evaluation, and local inference.

Variants

No	Variant	Cortex CLI command
1	Smollm2-1.7b	`cortex run smollm2:1.7b`

Use it with Jan (UI)

Install Jan using Quickstart
Use in Jan model Hub:
```
cortexhub/smollm2
```

Use it with Cortex (CLI)

Install Cortex using Quickstart
Run the model with command:
```
cortex run smollm2
```

Credits

Author: SmolLM2 Team
Converter: Homebrew
Original License: Apache 2.0
Papers: SmolLM2 Research

Downloads last month: 145

GGUF

Model size

2B params

Architecture

llama

Hardware compatibility

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

Paper for cortexso/smollm2

SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model

Paper • 2502.02737 • Published Feb 4, 2025 • 258