AdamHD: Decoupled Huber Decay Regularization for Language Model Pre-Training
Paper • 2511.14721 • Published • 1
🤝 Open to Collab
Kitsun
KitsuVp
·
AI & ML interests
None yet
Recent Activity
upvoted a paper 3 days ago
AdamHD: Decoupled Huber Decay Regularization for Language Model Pre-Training upvoted a paper 3 days ago
Correction of Decoupled Weight Decay upvoted a paper 3 days ago
Stochastic Rounding for LLM Training: Theory and PracticeOrganizations
upvoted 20
papers 3 days ago
Correction of Decoupled Weight Decay
Paper • 2512.08217 • Published • 1
Stochastic Rounding for LLM Training: Theory and Practice
Paper • 2502.20566 • Published • 1
The AdEMAMix Optimizer: Better, Faster, Older
Paper • 2409.03137 • Published • 7
Next-Latent Prediction Transformers Learn Compact World Models
Paper • 2511.05963 • Published • 6
Output Embedding Centering for Stable LLM Pretraining
Paper • 2601.02031 • Published • 1
MiLe Loss: a New Entropy-Weighed Loss for Mitigating the Bias of Learning Difficulties in Large Language Models
Paper • 2310.19531 • Published • 1
LAuReL: Learned Augmented Residual Layer
Paper • 2411.07501 • Published • 1
Attention Residuals
Paper • 2603.15031 • Published • 195
SimpleGPT: Improving GPT via A Simple Normalization Strategy
Paper • 2602.01212 • Published • 4
GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling
Paper • 2506.22049 • Published • 3
The Curse of Depth in Large Language Models
Paper • 2502.05795 • Published • 41
SeeDNorm: Self-Rescaled Dynamic Normalization
Paper • 2510.22777 • Published • 1
Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers
Paper • 2603.08343 • Published • 2
You Need Better Attention Priors
Paper • 2601.15380 • Published • 2
Group Representational Position Encoding
Paper • 2512.07805 • Published • 6
RePo: Language Models with Context Re-Positioning
Paper • 2512.14391 • Published • 14
Interleaved Head Attention
Paper • 2602.21371 • Published • 2
Token embeddings violate the manifold hypothesis
Paper • 2504.01002 • Published • 2
KHRONOS: a Kernel-Based Neural Architecture for Rapid, Resource-Efficient Scientific Computation
Paper • 2505.13315 • Published • 1