Transformer Architecture and Self-Attention Explained
Transformers are often described as if they are a mysterious reasoning machine. At the mechanical level, they are a repeated pattern: turn tokens into vectors, let each token look at other tokens through attention, transform those vectors, and predict the next token.
This guide is for developers who use LLMs but want the architecture vocabulary underneath them. You will trace one sentence through self-attention, understand why multi-head attention exists, and see why context length, position, and data quality still bound the system. For the broader model stack, connect this with Layers of AI and Embeddings Explained.
A Transformer Rewrites Token Vectors
A transformer does not read words the way a person reads a paragraph. It receives tokens, turns them into vectors, adds position information, and repeatedly rewrites each vector using information from the surrounding context. After many layers, the final vector at the current position is used to predict the next token.
The running example is The bank raised rates. The token bank can mean a financial institution or a river edge. Self-attention helps the bank vector absorb clues from raised and rates, so later layers carry a more context-aware representation.
Quick reference
- Token embeddings start as learned vector lookups, not dictionary definitions.
- Position information matters because attention alone does not know token order.
- Each layer rewrites vectors; it does not store a human-readable parse tree.
- The final prediction is a probability distribution over the vocabulary.
Remember this
A transformer is a stack of vector-rewriting layers: tokens become context-aware representations before the model predicts what token comes next.
Self-Attention Lets Tokens Compare Context
Self-attention gives each token three learned projections: query, key, and value. A query asks, "what information do I need?" Keys describe what each token can offer. Values carry the information that will be mixed into the output vector. The attention score between a query and keys decides how strongly one token uses another token's value.
In The bank raised rates, the bank query may attend strongly to rates. That does not prove the model understands finance the way a person does; it means the representation has learned statistical relationships that help the next-token objective. The mechanism is powerful because every token can condition on every earlier token in a causal language model.
Quick reference
- Queries and keys produce attention scores; values are the information mixed by those scores.
- Causal masks prevent a generation step from looking at future tokens.
- Attention is content-based routing, not a database lookup or symbolic proof.
- Longer context increases possible relationships but also increases compute and distraction.
Remember this
Self-attention routes information between token vectors by learned similarity, which lets one token's representation incorporate relevant context.
Multi-Head Attention Looks Through Several Lenses
A single attention pattern would force one kind of relationship to dominate. Multi-head attention runs several attention projections in parallel, so different heads can specialize in different relationships: local syntax, long-range reference, delimiters, code structure, or task formatting. The outputs are combined and passed through feed-forward layers.
Do not over-romanticize individual heads. Some heads are interpretable in narrow examples; many are distributed and hard to assign a clean role. The practical lesson is that transformer capacity comes from repeated composition: attention mixes information, feed-forward blocks transform it, and normalization/residual paths keep training stable.
Quick reference
- Heads are parallel attention mechanisms with separate learned projections.
- Feed-forward layers apply learned nonlinear transformations to each token vector.
- Residual connections preserve signal across deep stacks.
- Layer normalization helps keep activation scale trainable.
Remember this
Multi-head attention gives the model several relationship channels, while repeated layers compose those relationships into stronger token representations.
Failure Story: More Context Still Picks the Wrong Bank
Trigger. A prompt includes two paragraphs: one about river erosion and one about interest rates. Symptom. The model answers a question about bank using the financial meaning when the user meant the river edge. Root mechanism. Self-attention offers possible relationships; it does not guarantee the intended reference wins when context contains competing cues.
Recovery: rewrite the request with a clear referent, split unrelated context, and evaluate on examples with deliberate ambiguity. Prevention: when building RAG or agents, retrieve fewer, more relevant chunks and preserve section titles. More tokens can increase evidence, but they can also increase misleading associations.
Quick reference
- Ambiguity is not solved by attention alone; the input must make the intended referent recoverable.
- Competing context can pull attention toward the wrong clue.
- RAG quality matters because retrieved noise becomes part of the attention field.
- Practice: write two ambiguous prompts and compare answers after removing distracting context.
Remember this
Self-attention can use context, but it cannot guarantee the right context dominates when the prompt mixes competing evidence.
Key takeaway
Transformers work by repeatedly rewriting token vectors through attention and feed-forward blocks. The useful mental model is not magic reasoning; it is context-sensitive representation learning trained to predict tokens.
Practice (20 min): take one ambiguous sentence such as The bank was crowded. Add financial context, then river context, then both. Record how the answer changes. Pass when you can explain the behavior using tokens, context, attention, and competing cues rather than saying the model simply understands the word.
Related Articles
Explore this topic