How to Implement Backpropagation and AdamW Optimizer for LLMs Using CUDA and ROCm

A developer tutorial series on building a large language model training pipeline using HIP, CUDA, and ROCm has released its second installment. Part 2 focuses on implementing the backward pass, covering gradient computation for Linear layers, Softmax, and LayerNorm using custom GPU kernels. The guide also walks through coding an AdamW optimizer kernel designed to run efficiently across billions of model parameters on GPU hardware. Mixed-precision training using FP16 and BF16 formats is introduced to reduce VRAM usage on modern NVIDIA and AMD GPUs. The tutorial requires completion of Part 1, familiarity with calculus chain rule, and a GPU with at least 8GB of VRAM.