The ingenuity of the paper “Attention Is All You Need” is truly remarkable. After reading the paper and reviewing the GPT-2 implementation several times, I realized that a Transformer is really built on three fundamental components, each responsible for a different job.

Attention  → Route information
MLP        → Process information
Residual   → Enable deep learning

Miss any one of these, and modern LLMs would not exist.

1. Attention — Routing Information

The QKV attention mechanism is one of the most elegant ideas in deep learning.

Each token generates three vectors:

• Query (Q) — what information I’m looking for.
• Key (K) — what information I contain.
• Value (V) — the actual information I will provide.

Mathematically, Q and K are symmetric in the dot product—(QK^T) and (KQ^T) contain the same similarity information (up to transpose). Their roles are defined by interpretation rather than mathematics: Queries ask questions, while Keys answer them.

The dot product QK^T measures how “sticky” or similar two tokens are. After scaling by the square root of the key dimension, the scores are normalized using Softmax, producing attention probabilities. These probabilities are then used to compute a weighted average of the Values:

Empirically, the authors discovered that this simple formula works extraordinarily well.

2. Position Embeddings — Giving Words a Sense of Order

Attention alone doesn’t know whether a word appears at the beginning or end of a sentence.

Without positional information, ‘dog bites man’ and ‘man bites dog’ would look identical. The original Transformer solved this using sinusoidal positional embeddings, an incredibly clever design based on sine and cosine waves of different frequencies. Every position receives a unique pattern that can be added directly to the token embedding.

Later models, such as LLaMA and many modern LLMs, replaced absolute positional embeddings with RoPE (Rotary Position Embedding).

Instead of simply adding position vectors to token embeddings, RoPE rotates the Query and Key vectors before computing attention. This makes attention naturally aware of relative positions, allowing the model to better understand how far apart two tokens are.

For example,

The animal didn’t cross the street because it was tired.

and

The animal didn’t cross the street because it was busy.

may produce very different attention patterns, even though the word “it” appears in exactly the same position. This difference comes from the interaction between the contextual token representations and positional information carried into the attention computation.

(See my previous blog on RoPE for a deeper explanation.)

3. MLP — Processing Information

Attention gathers information, but gathering information isn’t enough.

The model still has to think about what it has collected.

That’s the job of the MLP (Multi-Layer Perceptron).

In GPT-2, the MLP looks like this:

Linear(C, 4C)
GELU
Linear(4C, C)

where C is the embedding dimension.

For GPT-2 Small:

768 → 3072 → 768

The intermediate expansion to 4× the embedding dimension dramatically increases the computational capacity available to each token.

This larger hidden space allows the model to perform much richer nonlinear transformations before compressing the representation back to its original size.

A useful way to think about it is:

• Attention communicates across tokens.
• The MLP thinks within each token.

Attention decides where to gather information. The MLP decides what that information means.

4. Residual Connections — Making Deep Networks Possible

Residual connections may be the simplest idea in the Transformer, yet they are absolutely essential.

x = x + Attention(...)
x = x + MLP(...)

Without these additions, every layer would completely overwrite the representation produced by the previous layer.

More importantly, during backpropagation, gradients would have to pass through dozens of nonlinear transformations, causing them to become smaller and smaller—a classic vanishing gradient problem that historically made very deep neural networks difficult to train.

Residual connections change the learning objective.

Instead of learning

Output = Entirely New Representation

each layer only needs to learn

Output = Input + Learned Change

In other words, every Transformer block learns a small correction rather than rebuilding everything from scratch.

This simple shortcut allows gradients to flow directly through hundreds of layers, making today’s extremely deep LLMs trainable.

Sometimes the most powerful ideas are also the simplest.

5. What Are the Billions of Parameters?

Every few months, a new model is released with more parameters than the previous generation.

But what exactly are those parameters?

After training, every parameter in the network becomes a fixed number.

They include:

• Token embedding (wte)
• Position embedding (wpe) or RoPE parameters
• Attention projection matrices (W_Q, W_K, W_V)
• Output projection (c_proj)
• MLP weights (c_fc, c_proj)
• LayerNorm scale and bias
• Language modeling head (lm_head)

Once training finishes, all of these weights are frozen.

Inference is simply a sequence of matrix multiplications using these fixed weights.

Nothing inside the model changes.

The only thing that changes is the input tokens.

Given the same prompt, the same model weights, and the same sampling settings, the forward pass is completely deterministic.

Final Thoughts

Looking back, it’s amazing how a handful of elegant ideas reshaped the entire AI industry.

• Attention routes information between tokens.
• Position embeddings (or RoPE) give the model an understanding of order and relative distance.
• The MLP performs rich nonlinear computation within each token.
• Residual connections make it possible to stack hundreds of layers without gradients disappearing.

Together, these components transformed deep learning.

Nearly every modern large language model—from GPT to LLaMA, Claude, Gemini, and many others—is still built on these same core principles introduced by Attention Is All You Need.

The architecture has evolved, scales have exploded from millions to trillions of parameters, and countless optimizations have been added. Yet at its heart, today’s most powerful AI systems are still following the same elegant blueprint proposed in that groundbreaking 2017 paper.