---
title: Positional Encoding
category: concepts
created: 2026-06-11
updated: 2026-12-20
tags: [positional-encoding, transformer, sequence-modeling, position-embeddings, sinusoidal-encoding, learned-embeddings, relative-position, mathematical-notation, position-matrix]
sources: [raw/feeds/2026-06-11-the-transformer-family-version-2-0.md]
confidence: high
---
# Positional Encoding
Mechanism in [transformer-architecture](/concepts/transformer-architecture) to inject information about token positions in sequences, compensating for the position-agnostic nature of attention mechanisms. Essential for maintaining sequence order information in models that process all positions simultaneously.
## Mathematical Framework
### Position Matrix
Following lilian-weng's notation:
$$\mathbf{P} \in \mathbb{R}^{L \times d}$$
Where each row $\mathbf{p}_i$ represents the positional encoding for input $\mathbf{x}_i$ at position $i$ in the sequence of length $L$.
### Integration with Input
Positional encodings are typically added to input embeddings:
$$\text{Input} = \mathbf{X} + \mathbf{P}$$
This preserves the original embedding dimension $d$ while injecting position information.
## Encoding Methods
### Sinusoidal Encoding (Original)
- Uses sine and cosine functions with different frequencies
- Allows extrapolation to longer sequences than seen in training
- Deterministic and doesn't require learning additional parameters
### Learned Embeddings
- Trainable position embeddings learned during model training
- Often performs better but limited to seen sequence lengths
- Requires additional parameters proportional to maximum sequence length
### Relative Position Encoding
- Encodes relative distances between positions rather than absolute positions
- More flexible for variable-length sequences
- Used in many modern transformer variants
## Key Properties
### Necessity in Transformers
- [attention-mechanisms](/concepts/attention-mechanisms) are inherently position-agnostic
- Without positional encoding, models cannot distinguish token order
- Critical for tasks where sequence order matters (language modeling, translation)
### Design Considerations
- Must be consistent across different sequence lengths
- Should not interfere with semantic information in embeddings
- Balance between expressiveness and computational efficiency
## See also
- [attention-mechanisms](/concepts/attention-mechanisms)
- [transformer-architecture](/concepts/transformer-architecture)
- lilian-weng