AI Systems Performance Engineering

• Chapter 1: Introduction and AI System Overview
• Chapter 2: AI System Hardware Overview
• Chapter 3: OS, Docker, and Kubernetes Tuning for GPU-Based Environments
• Chapter 4: Tuning Distributed Networking Communication

Resources

Books

• AI Systems Performance Engineering: Optimizing Model Training and Inference Workloads with GPUs, CUDA, and PyTorch (2025.12)
  • Code

Articles

• Making Deep Learning Go Brrrr From First Principles
• Hardware Architectures for LLM Inference

Talks

• The Engineering Behind Training a 2 Trillion Parameter LLM (2026.04)

GPU

• H100 Tensor Core GPU Architecture
• NVIDIA Blackwell Architecture Technical Brief
• NVFP4 Trains with Precision of 16-Bit and Speed and Efficiency of 4-Bit (2025.08)
• Using FP8 and FP4 with Transformer Engine
• NCCL and Communication Collectives
• NCCL Algorithms