Transformers, the tech behind LLMs | Deep Learning Chapter 5

What a transformer is, and what it is for

GPT stands for Generative Pre-trained Transformer: a bot that generates text, that learned from a massive amount of data (with room to fine-tune afterward), built on the transformer architecture.

The GPT acronym broken down; "Pre-trained" refers to the massive up-front training that a later fine-tuning step can build on. (0:16)

The original transformer (Google, 2017) was built for translation. The ChatGPT-style variant is trained instead to take a piece of text and predict the next chunk as a probability distribution over all possible continuations.

The whole task in one picture: given the text so far, put a probability on every possible next token. (3:19)

Prediction becomes generation by looping: sample a token from the distribution, append it, predict again. Run GPT-2 this way and the story wanders; swap in GPT-3 (same basic model, much bigger) and it suddenly makes sense. This repeated predict-and-sample loop is exactly what you watch when ChatGPT writes one word at a time.

How data flows through the model

At a high level, generating one word works like this:

1. The input is split into tokens (words, word-pieces, or common character combinations; for images or audio, small patches or chunks).
2. Each token becomes a vector (a list of numbers) meant to encode its meaning. Read as coordinates in a high-dimensional space, words with similar meanings sit close together.

Every token is turned into a vector. These columns of numbers are the model's working representation of the text. (3:57)

1. The vectors pass through an attention block, which lets them talk to each other and update their values based on context. Attention is what figures out that "model" in "a machine learning model" differs from "model" in "a fashion model".
2. They then pass through a multi-layer perceptron (MLP / feed-forward) block, where every vector is processed in parallel through the same operation, refining it further.
3. Attention and MLP blocks repeat, back and forth, until the meaning of the whole passage is baked into the last vector, which is turned into a probability distribution over next tokens.

The two workhorse blocks: attention (vectors exchange context) then a multilayer perceptron (each vector refined in parallel), stacked over and over. (4:40)

Every operation in both blocks is a giant pile of matrix multiplications. Understanding a transformer is mostly understanding how to read those matrices.

The shape and scale of the weights

Deep learning works because these models share one training algorithm (backpropagation) that scales well, provided the model follows a strict format: inputs are arrays of real numbers (tensors), transformed layer by layer, where parameters only ever touch the data through weighted sums (packaged as matrix-vector products).

The learned parameters are called weights. GPT-3 has 175,181,291,520 of them, organized into 27,938 matrices that fall into just 8 categories.

The scale, made countable: 175 billion weights, 27,938 matrices, 8 categories. GPT-3 stays the reference because later models' exact numbers are kept private. (11:14)

Keep a sharp mental line between the weights (the learned "brains" of the model, fixed after training) and the data being processed (whatever specific input is fed in for one run).

Embeddings: meaning as direction

The first matrix is the embedding matrix (W_E): one column per vocabulary word. Each column is the vector that word turns into. Its values start random and are learned from data.

The embedding matrix W_E: about 50,000 columns, one per token, each column the starting vector for that word. (13:08)

GPT-3's embeddings have 12,288 dimensions. Working in a space with that many distinct directions is what lets directions carry meaning.

A 3D projection of a much higher-dimensional space. During training the model settles on embeddings where directions carry semantic meaning. (14:33)

The classic demonstrationthe difference E(woman) − E(man) is close to E(queen) − E(king), so one direction in the space encodes something like gender.

E(woman) − E(man) is a direction. Add it to E(king) and you land near E(queen); "gender" is a direction the model discovered on its own. (15:09)

The same trick composesE(Hitler) + E(Italy) − E(Germany) lands near E(Mussolini); the model has learned directions for "Italian-ness" and for "WWII axis leader". (Grimmer example, same mechanism as the playful E(Germany) − E(Japan) + E(sushi) ≈ E(bratwurst).)

Directions compose. Country and role each have their own direction, so vector arithmetic maps one leader to another. (16:04)

A useful piece of intuition for the next chapter: the dot product of two vectors measures how well they align (positive when they point the same way, zero when perpendicular, negative when opposed), and it is itself a weighted sum, which fits the "everything is matrix multiplication" theme.

The embedding matrix is the first pile of weights: 50,257 tokens × 12,288 dimensions ≈ 617 million weights, a small slice of the eventual 175 billion.

Context: vectors that soak up their surroundings

Do not think of an embedding vector as fixed to one word. As it flows through the network it gets tugged by each block, so a vector that began as "king" can end up pointing at "a king who lived in Scotland, achieved his post by murdering the previous king, and is described in Shakespearean language." The point of the network is to let each vector absorb a far richer, context-specific meaning than a lone word could hold.

The model processes a fixed number of vectors at once, its context size (2048 for GPT-3). That limit is why long chats with early ChatGPT felt like the bot losing the thread.

The context window: the model can only hold this many token-vectors at once when predicting the next word. (19:55)

The output step: unembedding, softmax, temperature

The desired output is a probability distribution over next tokens. Two steps produce it.

First, the unembedding matrix (W_U) maps the last vector to one raw score for every token in the vocabulary. Give the model "Harry Potter … least favourite teacher, Professor ___" and a well-trained network assigns a high score to "Snape".

The unembedding step turns the final vector into one score per token; here "Snape" dominates at 0.78. (20:47)

Those raw scores are called logits. W_U mirrors the embedding matrix (rows instead of columns), adding another ~617 million weights.

Second, softmax turns the logits into a valid probability distribution (each value in [0,1], all summing to 1): exponentiate every number, then divide each by the total. The largest input ends up near 1, the rest near 0, but softly, so close-second choices still get real weight.

After softmax the scores are a probability distribution: mostly weight on the top candidate, some spread to the runners-up. (22:36)

A constant T (temperature) in the softmax exponents controls sharpness. Higher T spreads weight toward lower-probability tokens (more variety, more risk of nonsense); lower T concentrates on the top token; T = 0 always picks the maximum.

Temperature in action: at T = 0 the next word is "little" with 100% weight; at high T the distribution flattens and riskier words like "young" become likely. (24:37)

The chapter's real purpose is to lay the groundwork (embeddings, softmax, dot products, matrix multiplication) so the attention mechanism, the heart of the transformer, is smooth to learn in the next chapter.

Tools and resources

Claims to verify

• GPT-3 specifics stated as exact: 175,181,291,520 weights, 27,938 matrices, 8 categories, 12,288 embedding dims, 50,257-token vocabulary, 2,048 context size. Accurate for GPT-3; newer models differ and keep numbers private.
• "Invented in 2017 by Google for translation" (the "Attention Is All You Need" paper), correct.
• The E(queen) analogy is approximate: the video itself notes the true embedding of "queen" is farther off than the clean arithmetic suggests.

Glossary

• Token: the unit a transformer reads, a word, word-piece, or common character combination (for other modalities, an image patch or sound chunk).
• Embedding: the vector a token is turned into; a point in a high-dimensional space where direction can carry meaning.
• Embedding matrix (W_E): the learned matrix with one column per vocabulary token; the column is that token's starting vector.
• Unembedding matrix (W_U): the learned matrix that maps the final vector to one raw score per vocabulary token.
• Vector: an ordered list of numbers; here, the model's representation of a token's current meaning.
• Attention block: the operation that lets token-vectors exchange information so each one's meaning is updated by its context.
• Multi-layer perceptron (MLP) / feed-forward block: the operation that refines each vector in parallel, without vectors talking to each other.
• Weights: the model's learned parameters; the "brains" fixed after training, distinct from the data being processed.
• Tensor: the general term for the multi-dimensional array of real numbers the data is held in.
• Weighted sum: multiply each input by a parameter and add; the only way weights touch the data, usually packaged as matrix-vector products.
• Backpropagation: the training algorithm that adjusts weights and that scales well when the model follows the required array format.
• Dot product: multiply corresponding components of two vectors and sum; measures how aligned they are (positive/zero/negative).
• Context size: the fixed number of token-vectors the model can hold at once when predicting the next token (2,048 for GPT-3).
• Logits: the raw, unnormalized scores the unembedding step outputs, one per token, before softmax.
• Softmax: the function that turns a list of numbers into a probability distribution (each in [0,1], summing to 1) by exponentiating then normalizing.
• Temperature (T): a constant in the softmax exponents; higher T flattens the distribution (more variety), T = 0 always picks the top token.
• Probability distribution: the model's output, a weight on every possible next token that sums to 1.