Using Q, K, V in neural networks marks a significant advancement in recalling previous tokens or words within the context. However, challenges arise when relying solely on QKV, as exemplified by the indistinguishable phrases “a cat sits on the mat” and “a mat sits on the cat,” the latter being nonsensical. This indicates a need to address positional memory.

In the original transformer paper, the authors ingeniously introduced the sine/cosine rotation idea for positional labeling, though it proved impractical. In earlier iterations of GPT, up to GPT-2, a simple blending of token embeddings and positional embeddings was employed effectively for smaller contexts. For instance, in a context window of 1024 positions, the position ends at 1023 (starting from 0). However, this method becomes unsuitable for contexts longer than 1024 tokens.

RoPE is fundamentally similar to the original sine/cosine positioning in that it does not utilize pure embeddings but instead pairs them up. This results in two dimensions forming a rotational angle to indicate relative distances between positions, akin to using frequency. The key highlights are that RoPE not only overcomes the limitation of a fixed context size, such as 1024, but also acknowledges that positional information and token information are fundamentally related to each other relatively, rather than being absolute in position.

GPT-2: Position is an attribute of the token (“this is cat at position 17“). RoPE: Position is an attribute of the interaction between tokens. The rotation changes how Queries and Keys compare with one another, so positional information lives in the relationship between vectors rather than in an added embedding.