Singularity Systems: Zero to Hero

The Hacker's Guide to compiling PyTorch and CUDA

Singularity Systems: Zero to Hero is a direct follow up to Karpathy's Neural Networks: Zero to Hero. The textbook walks through, from scratch, how to build a CUDA C compiler picocuda that compiles llm.c and a PyTorch1 interpreter picograd that evaluates nanogpt.py.

Prerequisites

• Software 1.0: Discrete Mathematics (Graph Combinatorics)
• Software 2.0: Continuous Mathematics (Tensor Calculus)

Contributing

The textbook and its compiler implementations are opensource. To conribute or discuss, join the #singularity-systems work group in the GPU Mode Discord.

Citation

@article{j4orz2025singsys,
  author  = "j4orz",
  title   = "Singularity Systems: Zero to Hero",
  year    = "2025",
  url     = "https://j4orz.ai/zero-to-hero/"
}