LLM Architecture Explained: How Large Language Models Work
The infographic shows the right big shape: an LLM receives text, turns it into tokens, maps those tokens into vectors, runs transformer blocks, projects the final state to vocabulary probabilities, and emits generated text. The correction is small but important: the model does not literally understand words the way a person does. It learns statistical structure in token sequences and uses that structure to predict useful next tokens.
This guide is for developers who use LLM APIs and want the full request path in their head. We will follow one prompt, What is artificial intelligence?, through tokenization, embeddings, transformer blocks, output logits, and the autoregressive generation loop. For deeper zooms, connect this with LLM Tokenization Explained and Transformer Architecture and Self-Attention Explained.
The LLM Pipeline Has Six Moving Parts
An LLM request begins as text, but the model never consumes raw words. The application assembles the final prompt from system instructions, developer instructions, user text, retrieved context, and tool output. A tokenizer converts that final string into token ids. An embedding lookup turns each id into a vector. Transformer blocks repeatedly rewrite those vectors using attention and feed-forward layers. The output head projects the final vector into one score per vocabulary token. A decoder turns selected token ids back into text.
The running prompt is What is artificial intelligence?. The tokenizer might split it into pieces such as What, is, artificial, intelligence, and ?, depending on the model family. Those pieces become ids, then vectors. The model's job during generation is not to answer once; it predicts one next token, appends it to the context, and repeats.
Quick reference
- Text prompt: the exact rendered input after templates, memory, retrieved context, and user text are assembled.
- Tokenization: maps text pieces to integer vocabulary ids; token count is model/tokenizer-specific.
- Embeddings: learned vector lookups that place each token id into the model's hidden dimension.
- Transformer blocks: repeated attention, feed-forward, residual, and normalization operations.
- Output head: converts the final hidden state into logits and then a probability distribution.
Remember this
An LLM is an autoregressive token machine: text becomes ids, ids become vectors, vectors become next-token probabilities, and generation repeats.
Tokens Become Vectors Before Meaning Can Be Compared
Tokenization is a discrete boundary. The model sees ids such as 101, 2054, or 3201, not the string a user typed. The embedding matrix is a learned table where each token id retrieves a vector. If the model width is d_model = 4096, each token becomes a 4096-number representation before attention starts.
Those vectors are not dictionary definitions. They are coordinates learned during training because they helped predict surrounding tokens across massive text corpora. The vector for artificial carries patterns that tend to co-occur with words like intelligence, neural, machine, and synthetic. Position information is added too, because the same words in different order should not mean the same thing.
Quick reference
- A token can be a whole word, a subword, punctuation, whitespace, or byte-like fragment.
- The embedding matrix shape is roughly
vocabulary_size x d_model. - Position encodings or positional embeddings let the model distinguish order.
- Embeddings are learned for a specific model/tokenizer pair and are not interchangeable across models.
Remember this
Embeddings turn token ids into learned vectors, giving attention something numerical to compare and transform.
Transformer Blocks Rewrite Context
A transformer block has two main jobs. Self-attention lets each token representation mix information from other tokens in the context. Feed-forward layers apply learned transformations to each token vector after attention. Residual connections and layer normalization keep the repeated stack trainable and stable.
In the prompt What is artificial intelligence?, the representation for intelligence can attend to artificial, while the representation near the answer position can attend to the whole question. The block does not store a sentence diagram. It rewrites vectors so the next layer has a better context-aware representation. Modern LLMs repeat this block many times, which is why the infographic labels the transformer section as N x.
Quick reference
- Self-attention computes query/key/value projections and mixes value vectors by learned attention weights.
- Causal language models mask future positions, so generation cannot look at tokens that have not been produced yet.
- Multi-head attention gives the model several relationship channels in parallel.
- Feed-forward networks transform each token vector after context has been mixed.
- Add-and-normalize paths preserve signal across deep stacks and reduce training instability.
Remember this
A transformer block is a repeated vector-rewriting unit: attention mixes context, feed-forward layers transform it, and residual normalization keeps the stack usable.
The Output Layer Chooses From the Vocabulary
After the final transformer block, the model holds a context-aware vector for the current position. The output layer projects that vector into one score per vocabulary token. These raw scores are called logits. Softmax converts logits into a probability distribution, but the application usually does not take probabilities as the final answer; it samples or selects a next token according to decoding settings.
This is where product behavior changes. Low temperature makes the model prefer high-probability tokens and often gives steadier output. Higher temperature or nucleus sampling can make output more varied. The architecture produces probabilities; decoding policy decides how much randomness to allow.
Quick reference
- Logits are raw token scores before normalization.
- Softmax turns logits into probabilities over the full vocabulary.
- Greedy decoding picks the highest-scoring token; sampling can choose among likely alternatives.
- Temperature and top-p are decoding policies, not learned facts inside the model.
Remember this
The model scores every possible next token; the decoder turns those scores into one selected token at a time.
Generation Repeats Until a Stop Condition
The generated answer is not emitted as one complete paragraph internally. The model predicts a token, appends it to the context, then predicts the next token from the longer context. APIs may stream these tokens or token fragments to the user, which is why text appears gradually.
Stop conditions are part of the product contract: stop token, max output tokens, tool-call boundary, validation failure, moderation block, timeout, or user cancellation. This matters in production because a model that can keep generating also needs a budget, a cancellation path, and a validator when the output is supposed to follow a schema.
Quick reference
- Autoregressive generation means each new token becomes part of the next input.
- Streaming improves perceived latency but does not remove the need for output validation.
- Longer generated text costs more and can drift away from the original task.
- Structured outputs need validation after generation and sometimes bounded repair retries.
- For system design, track prompt tokens, completion tokens, latency, and stop reason together.
Remember this
Generation is a loop with stop conditions, budgets, and validation; without those controls, output quality becomes hard to operate.
Failure Story: A Definition Becomes Confident but Wrong
Trigger. The prompt asks, What is artificial intelligence?, but the surrounding context includes an outdated internal glossary that defines AI only as rule-based expert systems. Symptom. The model gives a confident answer that ignores machine learning and modern generative models. Root mechanism. The architecture predicts the next token from the context it receives; if the context is stale or misleading, attention can pull the answer toward the wrong source.
Recovery: remove the stale glossary, retrieve current source material, and ask the model to cite the evidence it used. Prevention: keep retrieved context narrow and versioned, log source ids, and evaluate the pipeline with changed-definition cases. The fix is not wishing the model understood better; it is controlling the evidence field the architecture can attend to.
Quick reference
- A fluent output can still be wrong when the prompt contains stale or conflicting evidence.
- Attention uses available context; it does not know which document your product intended unless you encode that boundary.
- RAG systems should filter by source version before ranking by semantic similarity.
- Practice cases should include conflicting definitions, stale docs, and missing evidence.
Remember this
LLM architecture makes context powerful, so production systems must treat context selection as a correctness boundary.
Key takeaway
The compact mental model is: prompt text becomes tokens, tokens become vectors, transformer blocks rewrite those vectors with context, the output head scores the vocabulary, and decoding emits one token at a time. That is enough architecture to reason about context windows, latency, cost, hallucination, streaming, and output validation without pretending the model is a symbolic database.
Practice (25 min): take the prompt What is artificial intelligence? and draw the request path with six boxes: text, tokens, embeddings, transformer stack, logits/softmax, generated text. Add one intentional failure: stale context, too many tokens, or invalid output format. Pass when you can point to the exact stage where the failure enters and the smallest guard that catches it. For production follow-ups, read Context Window Management for LLM Apps and LLM Streaming.
Related Articles
Explore this topic