Traditional neural networks could already generate language, but they had a fundamental weakness: they tended to forget earlier information as sequences became longer. The network compressed everything into hidden activations, making long-range dependencies difficult to learn. Although the learned 768-dimensional representations became highly effective, they were essentially a black box—there was no explicit mechanism for retrieving relevant information from earlier tokens.

The breakthrough of Attention Is All You Need was not simply adding “memory.” It introduced a structured way for the model to use its learned representations.

Each token is still represented by a 768-dimensional embedding, but instead of using one vector for every purpose, it is projected into three different roles:

• Query (Q) — What information am I looking for?
• Key (K) — Can I help answer that?
• Value (V) — What information should I provide if selected?

This separation transforms a passive embedding into an active retrieval system.

For every token, the model computes the similarity between its Query and every previous token’s Key:

Attention score = Query · Key

A higher score means the previous token is more relevant to the current token. After determining these relevance scores (followed by scaling and Softmax normalization), the model retrieves the corresponding Value vectors and combines them into a context-aware representation.

The elegance of this design is that retrieval is dynamic. Every token asks its own question and may retrieve different information from exactly the same sentence.

Another common misconception is that Q, K, and V assign fixed meanings to dimensions. They do not.

Researchers have found that certain directions within the embedding space correlate with concepts such as plurality, verb tense, sentiment, programming syntax, geographic relationships, and factual associations. However, there is no single dimension representing one concept—for example:

• Dimension 42 = gender
• Dimension 317 = city

Instead, meaning is distributed across many dimensions.

Q, K, and V learn different linear projections of this distributed representation.

• Query projection emphasizes features useful for searching.
• Key projection emphasizes features useful for being found.
• Value projection emphasizes features useful for passing information forward.

The same underlying embedding can therefore serve three completely different purposes depending on how it is projected.

Concretely,

H = Activation(XW + b)

For a single token:

h = Activation(xW + b)

The Transformer inserts a retrieval step before the usual neural network computation. Instead of immediately applying the activation function, every token is projected into three roles:

Q = XWQ
K = XWK
V = XWV

The model then measures how relevant every token is to every other token: Attention Scores = Q dot Kᵀ

Normalize the scores: Attention Weights = Softmax(Attention Scores / √dk)

Finally, retrieve information from the Value vectors: Output = Attention Weights × V

For one token, the output is simply a weighted average of every token’s Value vector:

Output(i) = Σ Attention(i,j) × Value(j)

Only after this retrieval does the Transformer continue with the familiar neural network computation:

Output = FeedForward(Output)
       = W₂ · Activation(W₁Output + b₁) + b₂