FareedKhan-dev / train-llm-from-scratch
A complete, from-scratch PyTorch implementation of the entire LLM training and alignment pipeline — no transformers, no trl, no peft — making the internals of modern LLM development fully transparent.
星标趋势
AI 分析
项目摘要
A comprehensive from-scratch implementation of a Transformer-based LLM training pipeline in pure PyTorch, covering everything from raw text tokenization through pretraining, SFT, reward modeling, DPO/PPO/GRPO alignment, and evaluation — all without relying on high-level libraries like transformers, trl, or peft.
为什么值得关注
With over 11K stars, this project stands out as an educational resource that demystifies the full LLM training stack by hand-writing every algorithm in plain PyTorch. It bridges the gap between understanding the 'Attention is All You Need' paper and producing a functional, aligned reasoning model on a single GPU.
优势
- End-to-end pipeline from raw text to aligned chat model with every algorithm hand-written in PyTorch
- Exceptionally clear README with visual pipeline diagrams, code structure breakdown, and step-by-step explanations
- Covers modern alignment techniques (SFT, DPO, ORPO, KTO, PPO, GRPO) in a single cohesive codebase
局限性
- No formal releases, Docker support, or example scripts, limiting reproducibility and production deployment
- Educational focus means it lacks optimizations (e.g., mixed precision, distributed training, memory efficiency) found in production frameworks
使用场景
- Learning and understanding the internals of Transformer training and alignment algorithms
- Prototyping small-scale LLM pretraining and fine-tuning experiments on a single GPU
- Serving as a reference implementation for implementing LLM training pipelines without heavy dependencies