README.md

---
license: mit
language:
- pt
pipeline_tag: text-generation
base_model:
- AxionLab-official/MiniBot-0.9M-Base
library_name: transformers
---

# 🧠 MiniBot-0.9M-Instruct

> **Instruction-tuned GPT-2 style language model (~900K parameters) optimized for Portuguese conversational tasks.**

[![Model](https://img.shields.io/badge/🤗%20Hugging%20Face-MiniBot--0.9M--Instruct-yellow)](https://huggingface.co/AxionLab-official/MiniBot-0.9M-Instruct)
[![License](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)
[![Language](https://img.shields.io/badge/Language-Portuguese-blue)](https://huggingface.co/AxionLab-official/MiniBot-0.9M-Instruct)
[![Parameters](https://img.shields.io/badge/Parameters-~900K-orange)](https://huggingface.co/AxionLab-official/MiniBot-0.9M-Instruct)

---

## 📌 Overview

**MiniBot-0.9M-Instruct** is the instruction-tuned version of [MiniBot-0.9M-Base](https://huggingface.co/AxionLab-official/MiniBot-0.9M-Base), designed to follow prompts more accurately, respond to user inputs, and generate more coherent conversational outputs in **Portuguese**.

Built on a GPT-2 architecture (~0.9M parameters), this model was fine-tuned on conversational and instruction-style data to improve usability in real-world interactions.

---

## 🎯 Key Characteristics

| Attribute | Detail |
|---|---|
| 🇧🇷 **Language** | Portuguese (primary) |
| 🧠 **Architecture** | GPT-2 style (Transformer decoder-only) |
| 🔤 **Embeddings** | GPT-2 compatible |
| 📉 **Parameters** | ~900K |
| ⚙️ **Base Model** | MiniBot-0.9M-Base |
| 🎯 **Fine-tuning** | Instruction tuning (supervised) |
| ✅ **Alignment** | Basic prompt-following behavior |

---

## 🧠 What Changed from Base?

Instruction tuning introduced significant behavioral improvements with no architectural changes:

| Feature | Base | Instruct |
|---|---|---|
| Prompt understanding | ❌ | ✅ |
| Conversational flow | ⚠️ Partial | ✅ |
| Instruction following | ❌ | ✅ |
| Overall coherence | Low | Improved |
| Practical usability | Experimental | Functional |

> 💡 The model is now significantly more usable in chat scenarios.

---

## 🏗️ Architecture

The core architecture remains identical to the base model:

- **Decoder-only Transformer** (GPT-2 style)
- Token embeddings + positional embeddings
- Self-attention + MLP blocks
- Autoregressive generation

No structural changes were made — only behavioral improvement through fine-tuning.

---

## 📚 Fine-Tuning Dataset

The model was fine-tuned on a Portuguese instruction-style conversational dataset composed of:

- 💬 Questions and answers
- 📋 Simple instructions
- 🤖 Assistant-style chat
- 🎭 Basic roleplay
- 🗣️ Natural conversations

**Expected format:**

```
User: Me explique o que é gravidade
Bot: A gravidade é a força que atrai objetos com massa...
```

**Training strategy:**
- Supervised Fine-Tuning (SFT)
- Pattern learning for instruction-following
- No RLHF or preference optimization

---

## 💡 Capabilities

### ✅ Strengths

- Following simple instructions
- Answering basic questions
- Conversing more naturally
- Higher coherence in short responses
- More consistent dialogue structure

### ❌ Limitations

- Reasoning is still limited
- May generate incorrect facts
- Does not retain long context
- Sensitive to poorly structured prompts

> ⚠️ Even with instruction tuning, this remains an extremely small model. Adjust expectations accordingly.

---

## 🚀 Getting Started

### Installation

```bash
pip install transformers torch
```

### Usage with Hugging Face Transformers

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "AxionLab-official/MiniBot-0.9M-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

prompt = "User: Me diga uma curiosidade sobre o espaço\nBot:"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=80,
    temperature=0.7,
    top_p=0.9,
    do_sample=True,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

### ⚙️ Recommended Settings

| Parameter | Recommended Value | Description |
|---|---|---|
| `temperature` | `0.6 – 0.8` | Controls randomness |
| `top_p` | `0.85 – 0.95` | Nucleus sampling |
| `do_sample` | `True` | Enable sampling |
| `max_new_tokens` | `40 – 100` | Response length |

> 💡 Instruct models tend to perform better at lower temperatures. Try values around `0.65` for more accurate and focused responses.

---

## 🧪 Intended Use Cases

| Use Case | Suitability |
|---|---|
| 💬 Lightweight Portuguese chatbots | ✅ Ideal |
| 🎮 NPCs and games | ✅ Ideal |
| 🧠 Fine-tuning experiments | ✅ Ideal |
| 📚 NLP education | ✅ Ideal |
| ⚡ Local / CPU-only applications | ✅ Ideal |
| 🏭 Critical production environments | ❌ Not recommended |

---

## ⚠️ Disclaimer

- Extremely small model (~900K parameters)
- No robust alignment (no RLHF)
- May generate incorrect or nonsensical responses
- **Not suitable for critical production environments**

---

## 🔮 Future Work

- [ ] 🧠 Reasoning-tuned version (`MiniBot-Reason`)
- [ ] 📈 Scaling to 1M–10M parameters
- [ ] 📚 Larger and more diverse dataset
- [ ] 🤖 Improved response alignment
- [ ] 🧩 Tool-use experiments

---

## 📜 License

Distributed under the **MIT License**. See [`LICENSE`](LICENSE) for more details.

---

## 👤 Author

Developed by **[AxionLab](https://huggingface.co/AxionLab-official)** 🔬

---

<div align="center">
  <sub>MiniBot-0.9M-Instruct · AxionLab · MIT License</sub>
</div>