Purpose
RNNs matter for two reasons. First, they were the canonical sequence model before transformers. Second, they make the core optimization problem of sequence learning painfully visible: both information and gradients must survive repeated application of the same transition.
The Elman Simple Recurrent Network
Elman’s simple recurrent network predicts the next symbol from the current input and a copy of the previous hidden state. At time step :
and for logits
The model’s bias is strong and clean:
- the same parameters are reused at every time step
- the hidden state is the memory
- prediction is a function of the current input plus that memory
Elman’s paper is important because it framed temporal structure as something the model could infer from raw sequence regularities rather than from manually defined symbolic units.
Backpropagation Through Time
Unroll the recurrence over steps and the RNN becomes a depth- network with shared parameters:
flowchart LR h0["h0"] --> c1["cell"] x1["x1"] --> c1 c1 --> o1["o1"] c1 -- "h1" --> c2["cell"] x2["x2"] --> c2 c2 --> o2["o2"] c2 -- "h2" --> c3["cell"] x3["x3"] --> c3 c3 --> o3["o3"] c3 -- "h3" --> etc["..."]
Every “cell” box is the same function with the same , , . The loop in the rolled form becomes depth in the unrolled form, and gradients must traverse that entire horizontal chain to reach early timesteps.
For total loss
the gradient to an early hidden state contains repeated Jacobian products:
This is the whole vanishing/exploding gradient story. If those Jacobians tend to shrink norms, the product vanishes. If they amplify norms, it explodes.
That is not a quirk of one optimizer. It is structural.
Vanishing gradients are a property of the parameterization
The product multiplies the same recurrent Jacobian times, so its norm behaves like for the dominant singular value . Unless sits almost exactly at 1, long-range gradient signal either dies or blows up exponentially in sequence length. Clipping handles the exploding side; the vanishing side needed an architectural fix, which is exactly what the LSTM’s additive cell-state update provides.
Why a Fixed-Length State Can Be a Bottleneck
Sequence-to-sequence encoder-decoder models before attention often forced the source sentence through one fixed-length vector. Bahdanau, Cho, and Bengio argue directly that this is a bottleneck. Their encoder-decoder extension lets the decoder search softly over relevant source positions rather than compressing the entire source into one vector once and for all.
That paper is the bridge between classic recurrent sequence models and the attention era.
LSTM
Hochreiter and Schmidhuber’s key idea is to create a path for constant error flow through the cell state. The paper explicitly says LSTM can bridge lags in excess of 1000 time steps by enforcing this constant error flow through constant error carousels.
One standard modern presentation is:
The important line is
because the additive state update is much friendlier to long-range credit assignment than repeatedly multiplying through one generic hidden transition.
The original paper also emphasizes locality: LSTM is local in space and time, with computational complexity per time step and weight.
GRU
GRUs compress the gating logic:
They are not in the source papers listed here, but the comparison is useful because it highlights the main architectural idea: make state updates additive and gate-controlled rather than purely recurrent and multiplicative.
Attention as the Escape Hatch
Bahdanau attention computes a context vector for decoder step :
with alignment weights
and score
where is the decoder state and is the encoder annotation at source position .
This is the moment sequence modeling changed. Instead of asking a fixed-width hidden state to remember everything, the model learned where to look.
NumPy Vanilla RNN
import numpy as np
def tanh(x):
return np.tanh(x)
class VanillaRNN:
def __init__(self, d_in, d_hidden, d_out, seed=0):
rng = np.random.default_rng(seed)
self.Wxh = rng.normal(0, 0.1, size=(d_hidden, d_in))
self.Whh = rng.normal(0, 0.1, size=(d_hidden, d_hidden))
self.bh = np.zeros(d_hidden)
self.Why = rng.normal(0, 0.1, size=(d_out, d_hidden))
self.by = np.zeros(d_out)
def step(self, x_t, h_prev):
h_t = tanh(x_t @ self.Wxh.T + h_prev @ self.Whh.T + self.bh)
o_t = h_t @ self.Why.T + self.by
return h_t, o_t
def forward(self, x):
B, T, _ = x.shape
h = np.zeros((B, self.Whh.shape[0]))
outputs = []
for t in range(T):
h, o = self.step(x[:, t], h)
outputs.append(o)
return np.stack(outputs, axis=1)PyTorch LSTM Language Model
import torch
import torch.nn as nn
class LSTMLM(nn.Module):
def __init__(self, vocab_size: int, d_model: int, n_layers: int = 2):
super().__init__()
self.embed = nn.Embedding(vocab_size, d_model)
self.rnn = nn.LSTM(
input_size=d_model,
hidden_size=d_model,
num_layers=n_layers,
batch_first=True,
)
self.lm_head = nn.Linear(d_model, vocab_size, bias=False)
def forward(self, token_ids: torch.Tensor) -> torch.Tensor:
x = self.embed(token_ids)
h, _ = self.rnn(x)
return self.lm_head(h)Executable Experiment
This companion trains a compact character-level GRU on a cropped WikiText-2 corpus. The loss curve, generated sample, and hidden-state behavior turn the recurrence equations into an inspectable training run.
Goal
Train a small character-level recurrent model on WikiText-2. We will crop the corpus down to a few hundred thousand characters so the notebook stays laptop-sized while still learning visible local structure.
For a recurrent network,
We will use a GRU because it keeps the code compact while still preserving the recurrent inductive bias.
## %pip install -q datasets torch matplotlib
import math
import random
import matplotlib.pyplot as plt
import torch
import torch.nn as nn
import torch.nn.functional as F
from datasets import load_dataset
torch.manual_seed(7)
random.seed(7)ds = load_dataset("Salesforce/wikitext", "wikitext-2-raw-v1")
train_text = "\n".join(ds["train"]["text"])[:300000]
valid_text = "\n".join(ds["validation"]["text"])[:50000]
vocab = sorted(set(train_text + valid_text))
stoi = {ch: i for i, ch in enumerate(vocab)}
itos = {i: ch for ch, i in stoi.items()}
train_ids = torch.tensor([stoi[ch] for ch in train_text], dtype=torch.long)
valid_ids = torch.tensor([stoi[ch] for ch in valid_text], dtype=torch.long)
print(f"chars={len(train_ids):,} vocab={len(vocab)}")chars=300,000 vocab=164
def get_batch(data: torch.Tensor, batch_size: int, block_size: int, device: str):
starts = torch.randint(0, len(data) - block_size - 1, (batch_size,))
x = torch.stack([data[s : s + block_size] for s in starts]).to(device)
y = torch.stack([data[s + 1 : s + block_size + 1] for s in starts]).to(device)
return x, y
class CharRNN(nn.Module):
def __init__(self, vocab_size: int, embed_dim: int = 64, hidden_dim: int = 128):
super().__init__()
self.embed = nn.Embedding(vocab_size, embed_dim)
self.rnn = nn.GRU(embed_dim, hidden_dim, batch_first=True)
self.proj = nn.Linear(hidden_dim, vocab_size)
def forward(self, idx, hidden=None):
x = self.embed(idx)
out, hidden = self.rnn(x, hidden)
logits = self.proj(out)
return logits, hidden
@torch.no_grad()
def generate(self, idx, max_new_tokens: int):
hidden = None
for _ in range(max_new_tokens):
logits, hidden = self(idx[:, -1:], hidden)
probs = logits[:, -1].softmax(dim=-1)
next_idx = torch.multinomial(probs, num_samples=1)
idx = torch.cat([idx, next_idx], dim=1)
return idxdevice = "cuda" if torch.cuda.is_available() else "cpu"
model = CharRNN(len(vocab)).to(device)
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-3)
batch_size = 64
block_size = 128
train_losses = []
for step in range(600):
xb, yb = get_batch(train_ids, batch_size, block_size, device)
logits, _ = model(xb)
loss = F.cross_entropy(logits.reshape(-1, logits.size(-1)), yb.reshape(-1))
optimizer.zero_grad()
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
optimizer.step()
train_losses.append(float(loss))
if step % 200 == 0:
print(f"step={step:04d} loss={float(loss):.4f}")step=0000 loss=5.1217
step=0200 loss=1.9348
step=0400 loss=1.7522
plt.figure(figsize=(8, 4))
plt.plot(train_losses)
plt.title("Char-RNN training loss")
plt.xlabel("step")
plt.ylabel("cross entropy")
plt.show()<Figure size 800x400 with 1 Axes>
start = torch.tensor([[stoi["T"]]], dtype=torch.long, device=device)
sample = model.generate(start, max_new_tokens=600)[0].tolist()
print("".join(itos[i] for i in sample))The mace often albwoun building texts . It . The Conduabor , and [ @,@ 000 god , 1s ) and won sertic gold dosent exteck operationsious lack tradicated by they was nubind muling continaliginal . The asiggued that worford opecting creatities , the muning its rifle thar , and withlin bypart an troingne of Aprubior , ifter activiled the shiption d. spection . He instinus mopernor plales , and Wather , ḏnyRual in an acterornarder by fuggulatur is reill . In and terall anaway . " A Griate recially sujical towner owerificating buildings tradition in the samited in the 72 @,@ 0000 been larnunes with su
What To Look For
The model usually learns punctuation, line breaks, and some short-range spelling statistics before it learns anything like semantics. That is exactly the point. The recurrent state is carrying local context forward step by step, and you can see how far that alone gets you.
Where RNNs Still Matter
- streaming settings where strict online updates are natural
- smaller-scale sequence problems
- domains where state compression is acceptable and low latency matters
Their core weakness remains the same: the hidden state is a narrow information bottleneck. Attention relaxes that bottleneck by turning memory access into a data-dependent retrieval problem.
Related Notes
- Feedforward Neural Networks
- Faster Causal Self Attention
- Neural Networks from Scratch
- Encoder-Decoder Transformers
- Decoder-Only Transformers