AliBuxdev/llm-query-classifier

🔍 LLM Query Classifier — General vs Real-Time

A fine-tuned NLP model that classifies whether a user query requires a static LLM response or a real-time data source. Built on top of MiniLM-L6 using Hugging Face Transformers.

---

💡 Why This Exists

Modern AI assistants face a core routing problem:

• "What is the capital of France?" → A static LLM can answer this perfectly.
• "What is the price of Bitcoin right now?" → A static LLM will hallucinate or give outdated info.

Without a classifier, you either always call an expensive real-time API (slow + costly), or you let the LLM guess (unreliable). This model solves that by routing queries intelligently before any expensive call is made.

---

🧠 Model & Approach

Detail	Value
Base Model	nreimers/MiniLM-L6-H384-uncased
Task	Binary Text Classification
Framework	Hugging Face Transformers + PyTorch
Training Set	~260 labeled queries
Validation Set	~63 queries (20% split)
Classes	0 = general, 1 = realtime

---

📊 Dataset

323 manually curated queries split across two classes:

• General (0): 145 examples — facts, definitions, history, science, explanations
• Real-time (1): 178 examples — current events, prices, weather, live scores, breaking news

Sample queries:

Query	Label
"Explain the theory of relativity"	general
"Who wrote Pride and Prejudice?"	general
"What is the current price of Bitcoin?"	realtime
"Latest news on the Ukraine war"	realtime
"Who is the Prime Minister of the UK right now?"	realtime

---

🚀 How to Use

1. Install dependencies

pip install transformers torch

2. Load and run the classifier

from query_classifier import QueryClassifier

classifier = QueryClassifier()

query = "What is the weather in Islamabad today?"
category, confidence = classifier.classify(query)

print(f"Category: {category}")       # realtime
print(f"Confidence: {confidence:.2f}")  # e.g. 0.97

3. Output

Category: realtime
Confidence: 0.97

---

📁 Project Structure

query-classifier/
├── train_classifier.py     # Fine-tuning script
├── query_classifier.py     # Inference class (plug-and-play)
├── training_data.csv       # Labeled dataset
├── trained_model/          # Saved model weights (after training)
│   ├── config.json
│   ├── tokenizer_config.json
│   └── model.safetensors
└── README.md

---

🏋️ Training

python train_classifier.py

Trains for 3 epochs with AdamW optimizer (lr=5e-5), saves the best checkpoint based on validation accuracy.

---

🔌 Integration Example

This classifier is designed to sit in front of your LLM pipeline:

category, confidence = classifier.classify(user_query)

if category == "realtime":
    response = call_search_api(user_query)   # Tavily, Serper, etc.
else:
    response = call_llm(user_query)          # GPT-4, Claude, etc.

---

🛠️ Built With

• Hugging Face Transformers
• PyTorch
• MiniLM — lightweight and fast

---

👤 Author

Built by [Your Name] — NLP & LLM Fine-tuning specialist. Open to freelance projects → [Your Upwork / LinkedIn link]