One realization keeps getting reinforced as I learn more about machine learning: the beauty of mathematics is that it reduces incredibly complex phenomena into symbols, linear algebra, and optimization. Language, vision, reasoning, code—even intelligence itself—are all eventually represented as vectors, matrices, and a sequence of computations.
The real ingenuity isn’t merely inventing formulas. It’s discovering a computational process that allows those formulas to solve complexity. Gradient descent and backpropagation are perfect examples. Rather than explicitly programming intelligence, we define a differentiable computation graph, measure how wrong the output is, and let optimization gradually discover the right parameters. Intelligence emerges not because we hand-crafted it, but because mathematics found a path through an unimaginably large search space.
As I work through Andrej Karpathy’s GPT implementation line by line, this realization appears repeatedly. Even something as conceptually rich as attention eventually boils down to a handful of linear algebra operations.
self.c_attn = nn.Linear(config.n_embd, 3 * config.n_embd)
One matrix multiplication simultaneously produces three different projections: Query, Key, and Value.
The output is simply sliced into three pieces, each corresponding to a different learned weight matrix.
q = x · Wqk = x · Wkv = x · Wv
Three projections from exactly the same input embedding.
At first, this seems arbitrary. Why three? Why not one?
The common explanation is intuitive:
- Query asks, “What am I looking for?”
- Key says, “What information do I contain?”
- Value carries the information to aggregate.
But after thinking about it longer, I realized something more interesting.
From a mathematical perspective, Query and Key are really symmetric counterparts.
Attention computes
QKᵀ
which is simply a bilinear interaction. The dot product itself has no notion of “asking” or “answering.” It merely compares two transformed vector spaces. During training, gradient descent discovers two projection matrices that minimize prediction error. Only afterward do we humans interpret one projection as “asking” and the other as “being matched.”
In other words, “What am I looking for?” and “What do I contain?” are useful mental models—not properties built into the mathematics.
The asymmetry emerges from optimization.
If Query and Key shared exactly the same projection matrix, attention would degenerate into a symmetric similarity matrix. Every relationship from token A to token B would mirror the relationship from B to A. Natural language is rarely that symmetric. Different learned projections allow attention to become directional, letting one token seek information differently from how another presents it.
Again, mathematics quietly reveals why the architecture works.
Even tensor reshaping tells a similar story.
k.view(B, T, n_head, head_size).transpose(1, 2)
At first glance this feels like mysterious tensor manipulation. In reality, no data is copied. .view() simply changes how memory is interpreted, while transpose() rearranges dimensions so that each attention head becomes an independent batch. Suddenly the GPU can compute every attention head in parallel.
Instead of one massive computation, we transform the problem into thousands of smaller matrix multiplications that GPUs excel at.
Everything is just changing perspectives.
The attention computation itself is remarkably elegant.
Q @ Kᵀ
produces raw similarity scores.
Scaling by √head_size prevents those scores from becoming excessively large and driving softmax into saturation.
The causal mask replaces future positions with negative infinity so autoregressive models cannot peek ahead.
Softmax converts arbitrary similarities into probabilities that sum to one, producing stable gradients and interpretable attention weights.
Finally,
Attention @ V
computes a weighted combination of value vectors.
Every output token simply becomes a weighted average of information gathered from other tokens.
Nothing magical.
Just linear algebra.
Another recurring lesson is that almost every architectural improvement is ultimately a trade-off.
Longer context windows let models reference information much farther away, improving document understanding. But computation grows quadratically with sequence length, increasing both memory usage and runtime. Efficient attention methods exist largely because this quadratic cost eventually becomes prohibitive.
Even batching follows the same mathematical philosophy. GPUs are designed for parallel matrix operations, so processing many sequences simultaneously keeps hardware fully utilized. Larger batches also average gradient noise, making optimization more stable, although they consume considerably more memory.
The same story repeats throughout deep learning history.
RNNs attempted to remember previous information sequentially.
LSTMs introduced forget gates and input gates, updating memory through simple additive equations.
CNNs discovered local spatial patterns using shared convolution kernels.
Then Transformers replaced recurrence almost entirely with attention.
Today Vision Transformers treat image patches exactly like language tokens. Swin Transformers improve efficiency through hierarchical windows. Yet CNNs remain competitive on mobile devices, small datasets, and latency-sensitive applications.
Every generation introduces a different mathematical abstraction for representing information.
Stepping back, I think this is what fascinates me most about AI.
We often describe models using human language—memory, attention, reasoning, understanding—but underneath, there are only vectors, matrices, probability distributions, and optimization. The “intelligence” we perceive emerges from the interaction of these mathematical objects after billions of gradient updates.
Perhaps that’s the real beauty of mathematics.
It doesn’t merely describe reality.
It compresses astonishing complexity into elegant abstractions and simple computations, allowing intelligence itself to emerge from nothing more than linear algebra and optimization.