Using Q, K, V in neural networks marks a significant advancement in recalling previous tokens or words within the context. However, challenges arise when relying solely on QKV, as exemplified by the indistinguishable phrases "a cat sits on the mat" and "a mat sits on the cat," the latter being nonsensical. This indicates a need … Continue reading Positioning Embedding in GPT-2 to RoPE

the next step is tracing what actually happens during a forward pass. Every token flows through a series of tensor transformations until it becomes the next-token prediction. Most of the code is simply reshaping tensors and making GPU computation efficient. Instead of computing Query, Key, and Value separately, GPT performs one large linear transformation: q, … Continue reading Grasping Codes and Math 02

One realization keeps getting reinforced as I learn more about machine learning: the beauty of mathematics is that it reduces incredibly complex phenomena into symbols, linear algebra, and optimization. Language, vision, reasoning, code—even intelligence itself—are all eventually represented as vectors, matrices, and a sequence of computations. The real ingenuity isn't merely inventing formulas. It's discovering … Continue reading Everything Eventually Becomes Math

It's an interesting journey to revisit Kaparthy's GPT-2 creation focusing on codes and math. One of the first pieces is understanding how automatic differentiation works. In a minimal neural network framework, every value keeps track of where it came from. The system remembers that out was created from self and other, so during backpropagation it … Continue reading Grasping Codes and Math 01

In the abstract, it claimes "KimiK3 is built on Kimi Delta Attention and Attention Residuals, which improves information flow sequence length and model depth. Together with Stable LatentMoE, which effectively activates approximately 2.5x improvement in overall scaling efficiency over Kimi K2." Their contributions are summarized as follows: Pre-training at the open frontier. We train a … Continue reading Take Away from KIMIK3:OPENFRONTIERINTELLIGENC

A common observation about coding agents is that their capability is the product of LLM × harness. The LLM provides intelligence. The harness provides everything around it—system prompts, workflows, memories, examples, MCP servers, retrieval, guardrails, tool orchestration. Early on, the harness mattered enormously because the models weren't yet good enough. Much of Claude Code's early … Continue reading Opus 5 Deleted 80% of the System Prompts

Traditional neural networks could already generate language, but they had a fundamental weakness: they tended to forget earlier information as sequences became longer. The network compressed everything into hidden activations, making long-range dependencies difficult to learn. Although the learned 768-dimensional representations became highly effective, they were essentially a black box—there was no explicit mechanism for … Continue reading Q, K, and V Structured Token’s Dimension Representation.

TermTerse explanation Sigmoid functionAn S-shaped function that maps any real number to a value between 0 and 1. Often interpreted as a probability. Logistic functionAnother name for the sigmoid function: σ(x)=11+e−x\sigma(x)=\frac{1}{1+e^{-x}}σ(x)=1+e−x1​. Activation functionA function applied to a neuron's weighted sum to produce its output. It introduces nonlinearity (e.g., sigmoid, ReLU, tanh). Loss functionMeasures the error … Continue reading Go Over Important Concepts

One of the most surprising developments in modern AI is that decoder-only models have largely dominated encoder-based models, even though both originated from the same landmark 2017 paper, Attention Is All You Need. Attention Is All You Need (2017) | ------------------- | | Encoder Decoder | | BERT GPT → Llama → Claude → DeepSeek … Continue reading Understanding Encoder vs Decoder: Why Decoder-Only Models Took Over LLMs

Policy answers: What action should I take? For example, in chess/go playing, the policy network takes the board as input and outputs probabilities. 1. GPT policy: next-token policy (simplest) During normal GPT pretraining, there is no reward or environment. It is supervised learning:predict next token\text{predict next token}predict next token The loss is cross-entropy:L=−log⁡P(correct token)L=-\log P(\text{correct token})L=−logP(correct token) The optimizer then adjusts … Continue reading Policy vs Optimizer