Baseten Engineer's PyTorch Walkthrough of 7 Years of Transformer Architecture Goes Viral

A Baseten inference engineer published a technical blog post tracing transformer architecture evolution from OpenAI's GPT-2 (2019) to Moonshot AI's Kimi K3, garnering 2.4 million views within days. The post uses runnable PyTorch code to explain each architectural iteration, focusing on how models store, update, and retrieve memory rather than simply scaling compute. A central theme is the KV cache — invented to avoid recomputing keys and values at every decoding step — which itself introduced a memory bandwidth bottleneck as sequence lengths grew. The post also distinguishes between linear attention, which replaces the softmax algorithm to reduce complexity from O(N²D) to O(ND²), and FlashAttention, which optimizes GPU memory access patterns without changing the underlying algorithm. It further covers DeltaNet, a 2024 approach that allows the model's state matrix to be updated rather than only appended to, tracing the idea back to Schmidhuber's 1990s fast weight programmers.