Diving deeper into Makemore codes to illustrate subtle details affecting the nn output. For example, the concept of "dead neurons" if the squashing function, say, tanh squashed too many inputs to the polar data points of -1 and +1, causing previous neuron's gradients killed: By resetting the scale of the weights and bias initialized, the … Continue reading Reproduce GPT2 (124M) by Andrej Karpathy 2 Weights and Bias Initials Normalization, BatchNorm and BackProp in Makemore

What is residual pathway? A residual pathway (or residual connection) is a mechanism in neural networks that allows the original input of a layer to be added directly to its output. It was introduced in ResNet (Residual Networks) and has since become a fundamental component in modern architectures like Transformers. In a Transformer block, the … Continue reading Some Neural Network Concepts

Tokenization is the process of breaking down text into smaller units (tokens) such as words, subwords, or characters. Different tokenization methods are used based on the task, language, and requirements of the model. Word-Based Tokenization Character-Based Tokenization, this increase computation cost a lot. Subword-Based Tokenization like Byte Pair Encoding (BPE), Wordpiece of BERT, SentencePiece. Sentence … Continue reading Reproduce GPT2 (124M) by Andrej Karpathy 3 Tokenization

""" you give this script some words (one per line) and it will generate more things like it. uses super state of the art Transformer AI tech this code is intended to be super hackable. tune it to your needs. Changes from minGPT: - I removed the from_pretrained function where we init with GPT2 weights … Continue reading Reproduce GPT2 (124M) by Andrej Karpathy 2 Makemore

Follow the github page by Dr. Karparthy: https://github.com/karpathy/build-nanogpt and youtube: https://www.youtube.com/watch?v=l8pRSuU81PU&list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ&t=1175s, the whole series is worth deep diving hence here it is, right from the very beginning. First, basics by Andrej, he illustrated how manually applied micro level backward propagation in the first great video: macrograd! with these two simple py file, engine.py and nn.py, … Continue reading Reproduce GPT2 (124M) by Andrej Karpathy 1 Micrograd

""" A top-level linear programming interface. .. versionadded:: 0.15.0 Functions --------- .. autosummary:: :toctree: generated/ linprog linprog_verbose_callback linprog_terse_callback """ import numpy as np from ._optimize import OptimizeResult, OptimizeWarning from warnings import warn from ._linprog_highs import _linprog_highs from ._linprog_ip import _linprog_ip from ._linprog_simplex import _linprog_simplex from ._linprog_rs import _linprog_rs from ._linprog_doc import (_linprog_highs_doc, _linprog_ip_doc, # noqa: … Continue reading Decompose Scipy Linprog.py

There are lot of buzz words about the birth of Genesis near end of 2024. However, diving deep into it, you would realize the essence of it is it's applying AI to solve complex high-dimensional physics problem that traditional numerical analysis fall short of. The big breakthrough they claim to realize 43 million fast frame … Continue reading Essence of Genesis AI

The next wave is to create lots of AI agents. So far the most successful ones are cursor and windsurf, the non-commercial part of windsurf is Codeium's plugin for Vim, open sourced on github, let's break it down and see if we can create such cool product too. under the autoload folder, there is the … Continue reading Codeium Plugin for Vim is Open Sourced?

AI Agents!

According to Perplexity, Genesis and Isaac are both platforms designed for robotics and AI development, but they have distinct features and focus areas: Genesis Platform Simulation Speed: Genesis can run simulations up to 43 million frames per second, which is 430,000 times faster than real-time7. Physics Engine: Integrates various state-of-the-art physics solvers into a unified … Continue reading Genesis and Isaac Platforms