In this blog, I'm gonna introduce some basic concepts about position embeddings, including absolute, relative and rotary position embeddings.
Transformers are foundational architecture of LLMs. However, they are position-agnostic. Therefore, positional information must be explicitly provided to help the model better understand word order in the sequence.