Function Description

Function

Perform rotary position embedding (RoPE) on key and query.

Formula

The core idea of RoPE is to embed positional information into vectors through rotation operations, rather than simply adding them. Specifically, for each position i and dimension d, the RoPE formulas are as follows:

Where,

• i is the position index in the sequence.
• d is the dimension index.
• d_model is the model dimension (usually the dimension of the embedding vector).
• cos and sin are cosine and sine functions to generate the rotation angle.