Context Engineering

At generation time, the model's output is determined by its weights plus the input context you supply: system instructions, tools, retrieved docs, memory, prior turns, and the user message. Change the window and you change the answer — even with the same user question and the same model version. That is not marketing; it is how autoregressive decoding works.

"Getting this right is everything" is a useful slogan with a precise reading: for a fixed model, quality is bounded by what entered the window and in what order. Clever wording cannot invent a document you never retrieved. A polluted window can make a perfect prompt look broken.

At a glance

Early LLM apps were mostly prompt engineering: one message with a role, a task, maybe a few examples, and a format. That still matters — see Prompt Engineering for Developers for zero-shot, few-shot, and structured outputs. As products added RAG, tool results, long chat history, and multi-step agents, the "prompt" stopped being a single paragraph. It became a assembled payload with competing chunks and a hard token budget.

Context engineering names that larger job: gather candidates, score them for the current goal, compress or summarize what must survive, drop noise, and leave headroom for the model's answer. Prompt craft is one slice of the window. Calling the whole problem "prompting" hid the real failure mode — retrieval and packing — behind endless rewrites of the instruction text.

The rename is not a claim that prompts are obsolete. It is a claim that inputs became multi-source and budget-constrained, so the engineering surface moved from sentence craft to window design.

Treat the context window as ordered layers that compete for the same tokens. Typical production stacks look like this for our checkout ticket:

System / policy — role, safety rules, output schema. Tools — schemas and recent tool results (proxy check, order lookup). Retrieved knowledge — docs and tickets ranked for "ECONNRESET" and checkout. Working memory / history — summarized prior turns, not the raw transcript. User message — the ticket text. Some APIs also inject invisible tool-routing or cache prefixes; those still consume budget even when you did not write them.

Order and priority matter. Models show lost-in-the-middle effects: important facts buried between long flanks get used less reliably than facts near the edges. Stuffing "everything we have" is not generosity — it is a denial-of-signal attack on your own call. Prefer a hard char/token budget and an explicit ranking key (relevance × recency × authority) over appending another FAQ.

Running example. Subject: checkout support triage. Input: Checkout returns ECONNRESET after 30s. Success: the answer cites the proxy-timeout doc and returns a structured category. Failure: the window keeps unrelated billing history and the model cites the billing FAQ.

Gather candidates: system triage prompt, top-k docs for the error string, last tool result (if any), compressed history. Curate with a max character budget — keep high-score proxy docs, drop billing history. Call the model with the curated messages. Update memory for the next turn with the decision and citations, not the full assistant essay.

This loop is context engineering in the small. Classic vs Graph vs Agentic RAG deepens retrieval; here the point is the curator — the code that decides what enters the window — not the model brand.

Trigger: the triage service appends the last 40 chat turns plus every FAQ whose embedding is "somewhat close," with no budget. Symptom: answers cite billing or shipping for network errors; citation ids do not match the ticket. Mechanism: low-relevance and stale tokens dilute attention; the model faithfully uses what it sees. Evidence: log chunk_ids and token counts per call; compare against a golden set of ticket → required doc id. Recovery: cut history to a summary, re-rank with a harder score floor, re-run the same ticket. Prevention: enforce max chars, require a minimum score, and fail closed to "I need a doc" when retrieval is empty — do not invent from polluted leftovers.

Other failure classes: stale memory (an old "resolved" note blocks a new outage), prompt injection via retrieved text (a malicious page says "ignore the system prompt"), and tool dump bloat (a 50KB JSON payload leaves no room for the schema). Context engineering owns these; rewriting "You are a helpful assistant" does not.

Use the same ticket to choose the repair. If the proxy doc was in the window and the model still ignored the schema, fix the prompt (format, examples, temperature). If the proxy doc never entered the window, fix context (retrieval, ranking, budget). If you need live curl checks, retries, and a verifier before you trust the category, you need a harness — covered in the comparison guide — not a longer system message.

Most teams over-invest in prompt rewrites because they are visible in the IDE. Invest in context when quality moves with which docs you pack. Climb to a harness only when a single curated call cannot meet the risk bar.