| | --- |
| | language: |
| | - en |
| | license: openrail |
| | tags: |
| | - diffusion-llm |
| | - parallel-generation |
| | - custom-transformer |
| | - cropmark |
| | datasets: |
| | - OpenAssistant/oasst1 |
| | metrics: |
| | - cosine_similarity |
| | base_model: |
| | - darwinkernelpanic/DiffReaper-5 |
| | --- |
| | |
| | # DiffReaper-5L |
| |
|
| | DiffReaper-5L is a **larger** version of DiffReaper-5, with **2048-dim embeddings** and a **24-layer Transformer**. |
| |
|
| | ## Model Details |
| |
|
| | - **Architecture:** 24-layer Custom Transformer with Time Embedding. |
| | - **Task:** Conditioned Text Diffusion (Prompt-Response). |
| | - **Training Objective:** Cosine Similarity Regression. |
| | - **Sampling:** 10-step iterative parallel denoising. |
| |
|
| | ## Usage (Inference) |
| |
|
| | To run inference: |
| |
|
| | ```python |
| | import torch |
| | # Assuming DiffReaperModel is defined as in train_diffreaper_5l.py |
| | |
| | model = DiffReaperModel(vocab_size=50257, n_embd=2048, n_head=32, n_layer=24).to("cuda") |
| | model.load_state_dict(torch.load("diffreaper5l_latest.pt")) |
| | model.eval() |
| | ``` |
| |
|
| | ## Fine-tuning |
| |
|
| | To fine-tune on a custom dataset, ensure your data loader provides **Prompt** + **Response** pairs. Use the same Cosine Similarity loss. |
