Purpose
These notes are about making large models run well. The recurring questions are simple: where the time goes, where the memory goes, and which optimization changes the real bottleneck. GPU basics and performance modeling are the best entry points because they make the later optimization notes easier to judge.
The rest of the section clusters around a few themes. Batching, parallelism, GPU interconnects, and memory management are system design questions. Optimizing GPU kernels, Triton, and how to write a fast kernel are implementation questions. Transformers and speculative decoding tie the systems work back to model structure. PEFT and preference optimization covers the adaptation side: how a base model gets fine-tuned and preference-tuned cheaply, and how the resulting adapters get served.
Entry points
- Foundations: GPU basics, performance modeling, roofline reference
- Scheduling and memory: batching, memory management, parallelism, GPU interconnects
- Kernel work: optimizing GPU kernels, Triton, how to write a fast kernel
- Model-specific techniques: transformers, quantization, speculative decoding, sparsity and pruning
- Fine-tuning and adaptation: PEFT and preference optimization
- Distributed training: distributed training of LLMs, distributed ML runtime architecture, mixture of experts