3. Scaling Networks

age of scaling (2020 - 2025)

Contents

• 3.1 Scaling Large Language Models
  • 1.1.2 Age of Scaling's Memory Movement Problem
• 3.2 Accelerating nanogpt with FlashAttention
• 3.3 Programming a Compiled and Distributed Tensor
  • 3.3.1 OpCode, OpNode Intermediate Graph Representation
  • 3.3.2 ExecItem Kernelizer/Fuser, Scheduler
  • 3.3.3 Runtime, Allocator, Heterogenous Runtime

3.1 Scaling Large Language Models

3.2 Accelerating nanogpt with FlashAttention

3.2 Programming a Compiled and Distributed Tensor

3.2.1 OpCode, OpNode Intermediate Graph Representation

3.2.2 ExecItem Kernelizer/Fuser, Scheduler

3.2.3 Runtime, Allocator, Heterogenous Runtime