From Dev.to
Rotary Position Embeddings: How Rotating Queries and Keys Encodes Relative Position
Rotary Position Embeddings: How Rotating Queries and Keys Encodes Relative Position Transformers have no recurrence and no convolution, so on their own they cannot tell whether a token came first, f...