Instructions to use midnightcoderagent/MidnightCoder-30B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use midnightcoderagent/MidnightCoder-30B with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf midnightcoderagent/MidnightCoder-30B
# Run inference directly in the terminal:
llama cli -hf midnightcoderagent/MidnightCoder-30B

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf midnightcoderagent/MidnightCoder-30B
# Run inference directly in the terminal:
llama cli -hf midnightcoderagent/MidnightCoder-30B

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf midnightcoderagent/MidnightCoder-30B
# Run inference directly in the terminal:
./llama-cli -hf midnightcoderagent/MidnightCoder-30B

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf midnightcoderagent/MidnightCoder-30B
# Run inference directly in the terminal:
./build/bin/llama-cli -hf midnightcoderagent/MidnightCoder-30B

Use Docker

docker model run hf.co/midnightcoderagent/MidnightCoder-30B

LM Studio
Jan

vLLM

How to use midnightcoderagent/MidnightCoder-30B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "midnightcoderagent/MidnightCoder-30B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "midnightcoderagent/MidnightCoder-30B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/midnightcoderagent/MidnightCoder-30B

Ollama
How to use midnightcoderagent/MidnightCoder-30B with Ollama:
```
ollama run hf.co/midnightcoderagent/MidnightCoder-30B
```
Unsloth Desktop

How to use midnightcoderagent/MidnightCoder-30B with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf midnightcoderagent/MidnightCoder-30B

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "midnightcoderagent/MidnightCoder-30B"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use midnightcoderagent/MidnightCoder-30B with Docker Model Runner:
```
docker model run hf.co/midnightcoderagent/MidnightCoder-30B
```

Lemonade

How to use midnightcoderagent/MidnightCoder-30B with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull midnightcoderagent/MidnightCoder-30B

Run and chat with the model

lemonade run user.MidnightCoder-30B-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use midnightcoderagent/MidnightCoder-30B with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf midnightcoderagent/MidnightCoder-30B

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default midnightcoderagent/MidnightCoder-30B

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use midnightcoderagent/MidnightCoder-30B with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf midnightcoderagent/MidnightCoder-30B

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "midnightcoderagent/MidnightCoder-30B" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Context Length increase

by MasterLooser - opened 8 days ago

Discussion

MasterLooser

8 days ago

Hey MidnightCoders,

2 Questions, the README talks about Quantisiation but theres only one model to download? Might want to delete that section from the Readme.

A personal request, since the model size is so small, would it be possible to increase the max token to 1m from the modelside? Ofc people who are hardware limited can only use whatever is possible for them, but if there isnt any downside to having such a high max value, why not increase it.

Best Regards,
MasterLooser.

midnightcoderagent

Owner about 16 hours ago

Hello MasterLooser!

Thank you for the suggestions!

You're absolutely right about the README. We are planning to release more quantized versions, but for now, it makes sense to adjust the README so it accurately reflects what is currently available.

And regarding your request: yes! We are going to create an 80B model with 1M context, as you suggested. It will be an interesting addition for users with enough hardware to take advantage of the larger context window.

One question, if you don't mind us asking: could you tell us a little about your hardware setup, friend? We'd love to know what kind of machine you're using to run MidnightCoder.

Thanks again for the feedback and the suggestion!

Best regards,
MidnightCoder Team

MasterLooser

about 6 hours ago

Hey Midnightcoders,

sounds amazing, not sure if the 80b can fit a 1m context in my vram but the 30b will def be able to :D

regarding hardware, i am just a bit of a nerd messing around with AI. I just got a 5090 and maybe get a DGX Spark or similiar in the future but atm just 32gb vram ^^

Best Regards,
MasterLooser.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment