Prerequisites

Math#

College-level mathematical maturity is essential for understanding the equations in research papers and textbooks. We want at least Calculus, Linear Algebra, and Probability. We might encounter other topics too. Pick one or more of the options below.

Grant Sanderson aka 3Blue1Brown has made a lot of great videos on various math topics. That said, I don’t think watching them can replace studying math more rigorously.

Part I of Deep Learning is a good refresher on the math, if you can handle it. (The rest of the book is a bit outdated these days.)

Books by Justin Skycak are a gentler introduction to various math topics:

• Introduction to Algorithms and Machine Learning
• Linear Algebra
• Calculus

Programming#

Being able to write and understand code by hand is still a vital skill in the age of AI.

It’s been a while since I learned to program. So I’m not sure how well the recommendations below work for people completely new to CS.

Harvard’s CS50 is probably a solid introduction to programming and computer science.

Spend extra time with Python, since most AI code is written in it. Maybe start with Kaggle’s Python tutorial.

[Optional] There’s a whole world of algorithmic programming (also known as data structures and algorithms, or DSA). Take a look at the famous LeetCode 75 problem set to get a taste.

Deep Learning#

Not strictly necessary, but it’s good to understand how deep learning models work. An LLM is technically just a supersized deep learning model.

Dive into Deep Learning by Aston Zhang et al. is, in my opinion, the best book introducing deep learning as a field. The first six chapters cover enough ground for our purposes. Note that the book doesn’t include much that was released after 2022.

Language Modeling#

Understand how a simple LLM like GPT-2 works. Pick any one of the options below.

Andrej Karpathy’s 1-hour Intro to Large Language Models video and 3.5-hour Deep Dive into LLMs like ChatGPT video.

[Paid] Sebastian Raschka’s Build a Large Language Model (From Scratch) book is a more detailed treatment of roughly the same topics.