```python # ============================================================ # 0. The key dimensions in GPT-2 Small # ============================================================ B = batch_size T = sequence_length C = 768 # token / residual representation dimension H = 12 # number of attention heads D = C // H # 64 dimensions per attention head # GPT-2 Small has: # # 12 Transformer … Continue reading Understanding GPT-2 from the Codes by Karpathy

What is a neuron? A neuron is a learned function that takes multiple input values, applies its own weights and bias, and produces one output value. In the simple house-price NN example, in GPT-2, each token's 768 numbers are fed into 3,072 neurons, producing 3,072 intermediate values, then compressed back to 768. overall, MLP parameters … Continue reading Crystal Clear about Dimensions in LLM Training

The ingenuity of the paper "Attention Is All You Need" is truly remarkable. After reading the paper and reviewing the GPT-2 implementation several times, I realized that a Transformer is really built on three fundamental components, each responsible for a different job. Attention → Route information MLP → Process information Residual → Enable deep learning … Continue reading Why “Attention Is All You Need” Changed AI Forever

Using Q, K, V in neural networks marks a significant advancement in recalling previous tokens or words within the context. However, challenges arise when relying solely on QKV, as exemplified by the indistinguishable phrases "a cat sits on the mat" and "a mat sits on the cat," the latter being nonsensical. This indicates a need … Continue reading Positioning Embedding in GPT-2 to RoPE

the next step is tracing what actually happens during a forward pass. Every token flows through a series of tensor transformations until it becomes the next-token prediction. Most of the code is simply reshaping tensors and making GPU computation efficient. Instead of computing Query, Key, and Value separately, GPT performs one large linear transformation: q, … Continue reading Grasping Codes and Math 02

One realization keeps getting reinforced as I learn more about machine learning: the beauty of mathematics is that it reduces incredibly complex phenomena into symbols, linear algebra, and optimization. Language, vision, reasoning, code—even intelligence itself—are all eventually represented as vectors, matrices, and a sequence of computations. The real ingenuity isn't merely inventing formulas. It's discovering … Continue reading Everything Eventually Becomes Math

It's an interesting journey to revisit Kaparthy's GPT-2 creation focusing on codes and math. One of the first pieces is understanding how automatic differentiation works. In a minimal neural network framework, every value keeps track of where it came from. The system remembers that out was created from self and other, so during backpropagation it … Continue reading Grasping Codes and Math 01

In the abstract, it claimes "KimiK3 is built on Kimi Delta Attention and Attention Residuals, which improves information flow sequence length and model depth. Together with Stable LatentMoE, which effectively activates approximately 2.5x improvement in overall scaling efficiency over Kimi K2." Their contributions are summarized as follows: Pre-training at the open frontier. We train a … Continue reading Take Away from KIMIK3:OPENFRONTIERINTELLIGENC

A common observation about coding agents is that their capability is the product of LLM × harness. The LLM provides intelligence. The harness provides everything around it—system prompts, workflows, memories, examples, MCP servers, retrieval, guardrails, tool orchestration. Early on, the harness mattered enormously because the models weren't yet good enough. Much of Claude Code's early … Continue reading Opus 5 Deleted 80% of the System Prompts

Traditional neural networks could already generate language, but they had a fundamental weakness: they tended to forget earlier information as sequences became longer. The network compressed everything into hidden activations, making long-range dependencies difficult to learn. Although the learned 768-dimensional representations became highly effective, they were essentially a black box—there was no explicit mechanism for … Continue reading Q, K, and V Structured Token’s Dimension Representation.