Vanilla Transformer

The original transformer-architecture introduced by Vaswani et al. in 2017 with the "Attention is All You Need" paper. Distinguished from later enhanced versions by its canonical encoder-decoder-models architecture commonly used in Neural Machine Translation (NMT) models. lilian-weng uses this term in her Version 2.0 survey to specifically refer to the foundational architecture before the emergence of simplified variants like BERT and GPT.

Architecture Overview

The vanilla Transformer establishes the foundational encoder-decoder pattern that became the template for subsequent transformer variants. This architecture was specifically designed for sequence-to-sequence tasks, particularly neural machine translation.

Key Components

1. Encoder-Decoder Structure: Full bidirectional encoder paired with autoregressive decoder
2. multi-head-attention: Parallel attention mechanisms for rich representation learning
3. positional-encoding: Sinusoidal position embeddings to capture sequence order
4. Feed-Forward Networks: Position-wise fully connected layers
5. Residual Connections: Skip connections with layer normalization

Historical Context

The vanilla Transformer served as the foundation for the transformer-family-evolution, spawning numerous architectural variants:

• Encoder-only models: BERT and variants focusing on bidirectional understanding
• Decoder-only models: GPT series optimized for autoregressive generation
• Specialized variants: Task-specific architectural modifications

Mathematical Foundation

Uses the comprehensive Mathematical Notation for Transformers established in Lilian Weng's Version 2.0 survey, providing precise mathematical definitions for all architectural components.

Significance

The vanilla Transformer's encoder-decoder architecture proved that attention mechanisms could replace recurrence entirely, enabling:

1. Parallelization: Simultaneous processing of sequence positions
2. Scalability: Foundation for large language model development
3. Transfer Learning: Pre-training paradigms for downstream tasks
4. Architectural Innovation: Template for specialized transformer variants

See also

• transformer-architecture
• encoder-decoder-models
• transformer-family-evolution
• Mathematical Notation for Transformers