Case study 02
GPTs
A decoder-only GPT written from scratch in PyTorch, about 10M parameters.
Overview
A from-scratch implementation of a GPT-style language model in roughly 200 lines of PyTorch. No Hugging Face, no shortcuts: tokenisation, attention, training loop and sampling are all written by hand.
It is trained on the tiny Shakespeare dataset and generates new text in the same style, character by character.
What it does
Multi-head self-attention
Causal masked attention with 6 heads per block, written from first principles.
Transformer blocks
6 stacked blocks with pre-norm LayerNorm, residual connections and feed-forward layers.
Positional embeddings
Learned token and position embeddings over a 256-token context window.
Regularisation
Dropout of 0.2 throughout to keep a small model from overfitting.
Training
AdamW over 5,000 iterations with periodic train and validation loss checks.
Generation
Autoregressive sampling that streams new Shakespeare-style text.
How it works
- 01Encode the corpus at character level and split into train and validation sets.
- 02Sample random 256-token blocks as training batches.
- 03Run them through embeddings, 6 transformer blocks and a language-model head.
- 04Optimise cross-entropy loss with AdamW.
- 05Sample from the trained model one token at a time.
Stack
- Python
- PyTorch
- Transformers
- Self-attention
- AdamW