Understanding Transformer Models: From Attention to BERT

Introduction

When Vaswani et al. published "Attention Is All You Need" in 2017, it quietly changed everything about how we approach sequence modeling. In this post, I'll break down the transformer architecture in a way that actually makes sense.

What's Wrong with RNNs?

Before transformers, we used RNNs (and their variants like LSTMs and GRUs) for sequential data. The problem?

• Sequential processing — can't parallelize
• Vanishing gradients — long-range dependencies are hard
• Slow to train — processing word-by-word

Enter: The Attention Mechanism

The core insight of transformers is self-attention — letting every token in a sequence attend to every other token simultaneously.

import torch
import torch.nn.functional as F

def scaled_dot_product_attention(Q, K, V):
    d_k = Q.size(-1)
    scores = torch.matmul(Q, K.transpose(-2, -1)) / (d_k ** 0.5)
    weights = F.softmax(scores, dim=-1)
    return torch.matmul(weights, V)

This single function is the heart of the transformer. For each token, we compute:

• Query (Q): What am I looking for?
• Key (K): What do I contain?
• Value (V): What do I output if selected?

Multi-Head Attention

Rather than one attention function, we run several in parallel:

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model, num_heads):
        super().__init__()
        self.heads = nn.ModuleList([
            AttentionHead(d_model, d_model // num_heads)
            for _ in range(num_heads)
        ])
        self.proj = nn.Linear(d_model, d_model)
    
    def forward(self, x):
        return self.proj(torch.cat([h(x) for h in self.heads], dim=-1))

Each head can learn different types of relationships (syntax, semantics, co-reference, etc.).

BERT: Bidirectional Pre-training

BERT (Bidirectional Encoder Representations from Transformers) took the transformer encoder and pre-trained it on two tasks:

1. Masked Language Modeling (MLM) — predict masked tokens
2. Next Sentence Prediction (NSP) — predict if two sentences are consecutive

The bidirectional context (unlike GPT's left-to-right) makes BERT exceptional for classification and extraction tasks.

Fine-tuning BERT

from transformers import BertForSequenceClassification

model = BertForSequenceClassification.from_pretrained(
    'bert-base-uncased',
    num_labels=3  # negative, neutral, positive
)

# Fine-tune on your dataset
trainer = Trainer(
    model=model,
    train_dataset=train_data,
    eval_dataset=val_data,
    compute_metrics=compute_metrics
)
trainer.train()

Conclusion

Transformers work because they:

1. Process all tokens in parallel (fast training)
2. Capture long-range dependencies via attention
3. Scale effectively with more data and compute

The attention mechanism is genuinely elegant — it lets the model learn which parts of the input matter for each prediction, rather than cramming everything into a fixed-size vector.

Next up: I'll cover how to implement a transformer from scratch in PyTorch.