Transformers Deep Dive

Explain the internals cold: positional encodings (RoPE), attention variants, BPE, KV cache, scaling laws. The T-shape depth vertical.

1. 1Positional encodings: from sinusoidal to RoPE4 q
2. 2Attention variants: MHA, MQA, and GQA4 q
3. 3Tokenization at scale: byte-pair encoding4 q
4. 4The KV cache and efficient inference4 q
5. 5Normalization and residuals in depth4 q
6. 6The full block, and a hard proof of causality4 q
7. 7Scaling laws4 q
8. 8Why transformers work4 q