Presentations in This Series
- SFT Pipelines →Datasets, chat templates, loss masking, sample packing; Axolotl, TRL, Unsloth, torchtune; eval harness; VRAM math; multi-GPU.
- LoRA & PEFT Variants →LoRA math and target modules; QLoRA NF4 + double quant; DoRA magnitude-direction; IA3; merging and multi-LoRA serving.
- RLHF & PPO →Reward modelling and Bradley-Terry; the PPO loop, KL penalty, clip ratio; failure modes; the three-model VRAM reality; TRL implementation.
- DPO and Cousins →DPO derivation and beta; IPO; KTO without pairs; ORPO without a reference; GRPO (DeepSeek-R1's group-relative variant); decision tree.
- Constitutional AI & RLAIF →Two-stage CAI, critique-revision loop, constitution authoring, RLAIF, helpfulness-vs-harmlessness-vs-honesty trade-offs.
Before fine-tuning: pretraining. How the base model these decks start from is trained in the first place (the loop and the loss, backpropagation, AdamW, schedules, mixed precision, memory, data and model parallelism, scaling laws) is the Transformer Training series (live), with an animated companion, the Training site (Foundations track, after Decoder); its last deck bridges to SFT, RLHF and DPO here.