inference-optimization/DeepSeek-V4-2.7B-tiny

DSV4-tiny-finetuned

This is a fine-tuned version of inference-optimization/DSV4-tiny-empty trained on famous internet copypastas.

Model Details

• Base Model: inference-optimization/DSV4-tiny-empty
• Architecture: DeepseekV4ForCausalLM
• Total Parameters: 2,689,440,743 (~2.7B parameters)
• Precision: bfloat16

Training Details

The model was fine-tuned using the training template from the create-tiny-model skill on 4 famous internet copypastas:

• Bee Movie aviation speech
• GNU/Linux interject copypasta
• FitnessGram Pacer Test
• Darth Plagueis the Wise

Training Configuration

• Target Perplexity: 3.0
• Batch Size: 2
• Learning Rate: 5e-5
• Max Steps: 1000 (early stopped at step 160)
• Training Runtime: 29.2 seconds
• Training Loss: 0.2243
• Final Perplexity: ~2.44 (achieved target)

Training Progress

The model achieved excellent convergence:

• Initial loss: 12.52
• Final loss: 0.000137 (at step 80)
• Training stopped early after consistently achieving target perplexity

Generation Example

During training validation, the model successfully generated:

Prompt: "According to all known laws" Output: "According to all known laws of aviation, there is no way a bee should be able to fly."

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("./DSV4-tiny-finetuned")
model = AutoModelForCausalLM.from_pretrained(
    "./DSV4-tiny-finetuned",
    device_map="auto",
    torch_dtype=torch.bfloat16
)

# Note: The model uses bfloat16 precision
# Ensure your inputs are properly cast to the correct dtype

Files

• config.json: Model configuration
• model.safetensors: Model weights (5.1GB)
• tokenizer.json: Tokenizer vocabulary
• tokenizer_config.json: Tokenizer configuration
• generation_config.json: Generation parameters
• training_args.bin: Training arguments used during fine-tuning

Notes

• The model was successfully fine-tuned and achieved the target perplexity of 3.0
• Training completed in under 30 seconds with early stopping at step 160
• The model memorized the training copypastas effectively, as evidenced by the low final loss
• Model uses DeepSeek V4 architecture with MoE (Mixture of Experts) layers