One realization keeps getting reinforced as I learn more about machine learning: the beauty of mathematics is that it reduces incredibly complex phenomena into symbols, linear algebra, and optimization. Language, vision, reasoning, code—even intelligence itself—are all eventually represented as vectors, matrices, and a sequence of computations. The real ingenuity isn't merely inventing formulas. It's discovering … Continue reading Everything Eventually Becomes Math

It's an interesting journey to revisit Kaparthy's GPT-2 creation focusing on codes and math. One of the first pieces is understanding how automatic differentiation works. In a minimal neural network framework, every value keeps track of where it came from. The system remembers that out was created from self and other, so during backpropagation it … Continue reading Grasping Codes and Math 01

In the abstract, it claimes "KimiK3 is built on Kimi Delta Attention and Attention Residuals, which improves information flow sequence length and model depth. Together with Stable LatentMoE, which effectively activates approximately 2.5x improvement in overall scaling efficiency over Kimi K2." Their contributions are summarized as follows: Pre-training at the open frontier. We train a … Continue reading Take Away from KIMIK3:OPENFRONTIERINTELLIGENC

A common observation about coding agents is that their capability is the product of LLM × harness. The LLM provides intelligence. The harness provides everything around it—system prompts, workflows, memories, examples, MCP servers, retrieval, guardrails, tool orchestration. Early on, the harness mattered enormously because the models weren't yet good enough. Much of Claude Code's early … Continue reading Opus 5 Deleted 80% of the System Prompts

Traditional neural networks could already generate language, but they had a fundamental weakness: they tended to forget earlier information as sequences became longer. The network compressed everything into hidden activations, making long-range dependencies difficult to learn. Although the learned 768-dimensional representations became highly effective, they were essentially a black box—there was no explicit mechanism for … Continue reading Q, K, and V Structured Token’s Dimension Representation.

TermTerse explanation Sigmoid functionAn S-shaped function that maps any real number to a value between 0 and 1. Often interpreted as a probability. Logistic functionAnother name for the sigmoid function: σ(x)=11+e−x\sigma(x)=\frac{1}{1+e^{-x}}σ(x)=1+e−x1​. Activation functionA function applied to a neuron's weighted sum to produce its output. It introduces nonlinearity (e.g., sigmoid, ReLU, tanh). Loss functionMeasures the error … Continue reading Go Over Important Concepts

One of the most surprising developments in modern AI is that decoder-only models have largely dominated encoder-based models, even though both originated from the same landmark 2017 paper, Attention Is All You Need. Attention Is All You Need (2017) | ------------------- | | Encoder Decoder | | BERT GPT → Llama → Claude → DeepSeek … Continue reading Understanding Encoder vs Decoder: Why Decoder-Only Models Took Over LLMs

Policy answers: What action should I take? For example, in chess/go playing, the policy network takes the board as input and outputs probabilities. 1. GPT policy: next-token policy (simplest) During normal GPT pretraining, there is no reward or environment. It is supervised learning:predict next token\text{predict next token}predict next token The loss is cross-entropy:L=−log⁡P(correct token)L=-\log P(\text{correct token})L=−logP(correct token) The optimizer then adjusts … Continue reading Policy vs Optimizer

MilestoneBig ideaAlexNet (2012)Learn from labeled examples (supervised learning).ResNet (2015)Train very deep networks effectively.AlphaGo Zero (2017)Learn through self-play with reinforcement learning.Transformer (2017)Replace recurrence with attention.Vision Transformer (2020)Apply Transformers directly to images. The evolution of modern AI can be understood as a series of breakthroughs in how machines represent information, learn patterns, and make decisions. 1. LeNet: … Continue reading From CNN to Transformer: How AI Learns to See and Reason

Abstract "We introduce Kimi K2.5, an open-source multimodal agentic model designed to advance general agentic intelligence. K2.5 emphasizes the joint optimization of text and vision so that two modalities enhance each other. This includes a series of techniques such as joint text-vision pre-training, zero-vision SFT, and joint text-vision einforcement learning. Building on this multimodal foundation, … Continue reading Read Paper: KIMIK2.5: VISUAL AGENTIC INTELLIGENCE