Large language models (LLMs) can produce impressive results from a single prompt. However, building a reliable AI application usually requires much more than writing a good instruction. A production application may need to retrieve information from documents, remember relevant details from earlier interactions, choose tools, preserve conversation history, and decide which information should be included in the model’s next request.

This broader discipline is known as context engineering.

Context engineering is the practice of designing a system that gives an LLM the information and tools it needs for the current step of a task. The goal is not to provide as much information as possible. The goal is to provide the right information, at the right time, in the right format.

In this tutorial, we will develop an intuition for context engineering, understand how it differs from prompt engineering, explore its main building blocks, and see why carefully selected context can improve both response quality and efficiency.

1. From Prompt Engineering to Context Engineering

Prompt engineering focuses on the instructions and input we give to a model. For example, we might ask an LLM to:

You are a helpful technical support assistant.
Explain the difference between an API key and an access token.
Keep the explanation beginner-friendly.

This is useful, and prompt design remains important. However, a real application often needs to do more than send one carefully written prompt.

Imagine building a support assistant for a software product. A user asks:

What is the rate limit for the API endpoint I am using?

To answer correctly, the model may need:

• The product’s current API documentation.
• The specific endpoint the user is referring to.
• Details from the current conversation.
• The user’s account or plan information, if the application is authorized to use it.
• A tool that can look up current account-specific limits.

A static prompt cannot reliably contain every possible document, user detail, and live value. The application needs a system that finds and assembles relevant context for each request.

That is the shift from thinking only about a prompt to designing the broader context supplied to the model.

The second approach is dynamic. The context can change from one step to another depending on what the user asks and what the application discovers.

Prompt engineering and context engineering are complementary. A well-designed prompt tells the model how to behave. Context engineering determines what useful information and capabilities the model should have available when it follows those instructions.

2. What Is Context Engineering?

Andrej Karpathy described context engineering as the careful practice of filling the context window with the right information for the next step.

The important phrase is for the next step. An AI application should not blindly send every piece of information it has. It should decide what the model needs to know now.

A practical definition is:

Context engineering is the discipline of designing systems that provide LLMs with the right information and tools, at the right time, in a useful format, so they can complete the current task reliably.

This definition contains four important ideas.

2.1 Dynamic systems

Context engineering is not simply manual prompt writing. It involves building application logic that can retrieve, select, update, and organize context automatically.

For example, a documentation assistant might retrieve different document sections depending on the question. A coding agent might inspect files only when it needs them. A customer-support assistant might retrieve account details only when the user’s request requires them and the application has permission to access them.

The context is assembled as the task progresses.

2.2 The right information

More information does not automatically produce a better answer.

Suppose a user asks for the API rate limit. Sending the entire product manual, including installation instructions, UI documentation, troubleshooting guides, and unrelated examples, adds little value.

The relevant rate-limit section is much more useful. The objective is to maximize relevance, not volume.

2.3 The right time

Information should be provided when it is needed.

For example, an agent does not need to retrieve a user’s billing history before it knows that the task involves billing. It may first understand the request, then retrieve the relevant billing record if required.

This avoids unnecessary retrieval and keeps the context focused.

2.4 The right format

The same information can be presented in different ways. A model may work more reliably when the information is clearly structured and labeled.

Compare an unstructured note:

The basic plan has a limit. The professional plan has a different
limit. Some endpoints have separate limits. Limits can depend on
the account and may change.

With a structured representation:

Product: Example API

Rate limits:
- Basic plan: See current account configuration.
- Professional plan: See current account configuration.
- Endpoint-specific limits may apply.

Important:
- Use the current account configuration as the source of truth.
- Do not assume all endpoints share the same limit.

The second representation makes relationships and qualifications easier to identify. The application should choose a format that fits the information and the task.

3. The Context Window as Working Memory

A useful way to understand an LLM’s context window is to compare it with working memory.

When solving a problem, a person can actively consider only a limited amount of information at once. If someone gives us a large collection of unrelated documents, finding the few important facts becomes difficult. A short, relevant summary may be more useful.

An LLM also has a finite context window. The context window is the amount of input and generated content the model can handle in a particular interaction, subject to the model’s limits.

It may contain:

• System and developer instructions.
• The current user request.
• Recent conversation messages.
• Retrieved document passages.
• Summaries or memory.
• Tool descriptions and tool results.
• The model’s own intermediate or previous outputs, depending on the application design.

The context window is not the same thing as persistent memory. Information included in one request is available to the model for that request, but it is not automatically stored for future conversations.

3.1 Every token has a cost

Eugene Yan describes context as a budget. This is a useful engineering perspective because every additional token has potential costs.

Financial cost: Many hosted model APIs charge based on the number of input and output tokens. Sending unnecessary context can increase the cost of each request.

Processing cost: More input generally requires more computation and can increase latency. The exact relationship depends on the model, serving system, and attention implementation.

Attention and relevance: A large context can contain competing or distracting information. Important details may be harder for the model to use consistently, especially when the context is poorly organized or contains conflicting material.

This does not mean that every long prompt is bad. Some tasks genuinely require large amounts of source material. The goal is to include what the task needs while avoiding irrelevant or redundant content.

A useful principle is:

Maximize useful information per token, not the number of tokens in the context window.

3.2 A simple signal-versus-noise example

Imagine asking a model to answer a question about an API’s rate limits.

The application has a 50-page product manual. Only one short section explains the relevant rate limits. Sending the entire manual may work, but it introduces unnecessary information. A context-engineered application identifies the relevant section and supplies that section with the question.

The model receives less material, but the material is more useful for the task.

4. Five Core Techniques of Context Engineering

A practical context-engineering system commonly combines several techniques. Five important building blocks are:

1. Retrieval-Augmented Generation (RAG).
2. Memory.
3. Selective tool use.
4. Context assembly.
5. Context compression.

These techniques solve different problems. They are not necessarily separate services; an application may implement several of them in one orchestration layer.

4.1 Retrieval-Augmented Generation (RAG)

RAG allows an application to retrieve relevant information from an external knowledge source and include it in the model’s context.

This is useful when the answer depends on information that is not reliably available in the model’s parameters, such as:

• Internal company documentation.
• Product manuals.
• Technical specifications.
• Knowledge-base articles.
• Frequently changing policies.
• A collection of domain-specific documents.

Retrieval can use keyword search, semantic search with embeddings, or a hybrid approach. The appropriate method depends on the data and the query.

The important context-engineering decision is not merely whether retrieval exists. It is deciding which passages are relevant, how many to include, how to handle conflicting sources, and how to present the retrieved material.

4.2 Memory

Memory allows an application to preserve useful information beyond the immediate request.

For example, a learning assistant might retain that a user is studying neural networks and prefers explanations that introduce intuition before mathematics. A project assistant might preserve the current project’s goals, decisions, and unresolved tasks.

Memory helps avoid asking for the same information repeatedly. However, not every message should become a persistent memory. Applications should decide what is useful to retain, how long to retain it, and when it should be updated or removed.

A useful conceptual hierarchy has three levels:

This is a conceptual model, not a universal memory architecture. Different applications may use different storage systems and retention policies.

Working memory is the information currently supplied to the model. It may include the latest messages, instructions, retrieved passages, and tool results.

Short-term memory can preserve a summary of a session or an ongoing task. Instead of resending a long conversation, the application can include a concise summary of the important decisions and current state.

Long-term memory stores selected information that may be useful across sessions. It might be held in a database, document store, or another persistence layer. When relevant, the application retrieves only the appropriate facts and adds them to the current context.

The central lesson is that memory storage and context selection are different operations. A system may store many facts, but it should retrieve only the facts that help with the current task.

4.3 Selective tool use

Tools allow an LLM application to interact with external systems. Depending on the application, tools might:

• Query a database.
• Search documentation.
• Read a file.
• Call an external API.
• Create a support ticket.
• Execute a calculation.
• Retrieve current account information.

Giving a model access to many tools does not mean every tool should be used for every request.

Consider a user asking:

What is the current status of order 5821?

A useful application may need an order-lookup tool. It probably does not need a weather tool, a document-generation tool, or a general analytics tool.

A selective tool-use flow looks like this:

Tool selection should consider relevance, permissions, reliability, and the consequences of the action. A tool that changes external state—such as placing an order or deleting a record—requires suitable authorization and safeguards.

The context-engineering goal is to make the right capabilities available without overwhelming the model with unnecessary tool descriptions or irrelevant results.

4.4 Context assembly

Context assembly is the process of bringing the selected information together into the model’s next input.

It may combine:

• Stable instructions.
• The current user request.
• Relevant conversation history.
• Retrieved document passages.
• Selected memory.
• Tool results.
• Task-specific constraints and output requirements.

A simple conceptual example:

The order and structure can matter. For example, an application may place the user’s current request near the end of the assembled input, label retrieved passages as reference material, and clearly distinguish tool output from instructions.

Context assembly also needs to handle conflicts. If an older conversation summary disagrees with a current authoritative database result, the application should have a clear source-of-truth policy rather than simply placing both facts into the prompt without explanation.

4.5 Context compression

Context compression reduces the amount of information that must be passed to the model while preserving what is important for the task.

Common approaches include:

• Removing duplicate passages.
• Keeping only the most relevant document sections.
• Summarizing older conversation messages.
• Replacing lengthy tool output with a concise result.
• Extracting structured facts from unstructured text.
• Removing irrelevant metadata and boilerplate.

For example, a long conversation may contain many messages about a project. The next task may require only the goal, key decisions, current status, and next action.

Compression must be careful. If a summary removes an important exception, date, condition, or qualification, the model may produce an incorrect answer. For high-stakes or detail-sensitive tasks, it may be better to retrieve the original source rather than rely only on a summary.

5. Putting the Techniques Together

The five techniques work together in a dynamic application.

Imagine a technical-support assistant. A user asks:

I am getting a rate-limit error. What should I do?

A context-engineered application might proceed as follows:

1. Understand the request. Identify that the user needs help with an API rate-limit error.
2. Retrieve knowledge. Use RAG to find the relevant rate-limit documentation and troubleshooting instructions.
3. Select memory. Include a relevant detail from the current session, such as the endpoint the user has already mentioned.
4. Select tools. If the user asks about their current account limit, use an authorized account-lookup tool.
5. Assemble context. Combine the relevant documentation, account result, recent conversation, and response instructions.
6. Compress if necessary. Remove duplicate or unrelated content while preserving important conditions.
7. Generate the response. Ask the model to explain the likely cause and recommend the appropriate next step.

The system does not have to perform every operation for every request. If a question can be answered from the current conversation, it may not need RAG or an external tool. Context engineering includes knowing what not to retrieve or include.

6. Bad Context Engineering: Sending Everything

Let’s compare two approaches using a small product-documentation example.

Suppose a product has several documents:

Document 1: Product overview
Document 2: Installation guide
Document 3: Authentication guide
Document 4: API rate limits
Document 5: Error handling
Document 6: SDK examples

The user asks:

What are the API rate limits?

A naive approach may concatenate all available documentation and send it to the model.

all_documentation = "\n".join(documents)

prompt = f"""
Answer the user's question using the documentation.

Documentation:
{all_documentation}

User question:
{user_query}
"""

This is easy to implement, but it has several disadvantages.

Higher token usage: The model receives documents unrelated to the question.

Potentially higher cost: If the API charges for input tokens, sending unnecessary content increases the input-token cost.

Potentially higher latency: Processing a larger input can take longer, depending on the model and serving system.

More distracting information: The relevant rate-limit information is mixed with installation instructions, authentication details, and unrelated examples.

Less predictable behavior: The model may overlook a qualification or focus on an unrelated passage, especially when the supplied material is poorly organized or contains conflicting statements.

The problem is not that large contexts always fail. The problem is that the application has made no attempt to identify what the current task actually requires.

7. Good Context Engineering: Retrieve Only Relevant Information

A better approach is to retrieve the parts of the documentation that match the user’s question.

For a small demonstration, we can use simple keyword matching. This is not a production-grade RAG implementation, but it makes the context-selection process easy to understand.

Assume that documents are represented as records:

documents = [
    {
        "id": "doc-1",
        "section": "Overview",
        "content": "The product provides APIs for managing projects."
    },
    {
        "id": "doc-2",
        "section": "Authentication",
        "content": "API requests must include a valid API key."
    },
    {
        "id": "doc-3",
        "section": "Rate limits",
        "content": "Requests are subject to endpoint-specific rate limits. "
                   "The current account limit is shown in the account dashboard."
    },
    {
        "id": "doc-4",
        "section": "Installation",
        "content": "Install the SDK using the package manager."
    }
]

We can create a simple keyword-based retrieval function:

def retrieve_documents(query, documents, top_k=3):
    query_terms = set(query.lower().split())
    scored_documents = []

    for document in documents:
        content = (
            document["section"] + " " + document["content"]
        ).lower()

        score = sum(
            1 for term in query_terms if term in content
        )

        if score > 0:
            scored_documents.append((score, document))

    scored_documents.sort(
        key=lambda item: item[0],
        reverse=True
    )

    return [
        document
        for score, document in scored_documents[:top_k]
    ]

We can then retrieve the most relevant documents and build the context:

user_query = "What are the API rate limits?"

retrieved_documents = retrieve_documents(
    user_query,
    documents,
    top_k=3
)

context = "\n\n".join(
    f"Section: {document['section']}\n"
    f"Content: {document['content']}"
    for document in retrieved_documents
)

prompt = f"""
Answer the question using the reference material below.
If the material does not contain the answer, say so.

Reference material:
{context}

Question:
{user_query}
"""

The important difference is that the model receives selected reference material rather than the entire document collection.

This keyword-based function is intentionally simple. It does not understand synonyms, semantic similarity, document structure, or complex query intent. A production retrieval system may use embeddings, vector search, hybrid retrieval, reranking, metadata filters, and other techniques.

The example demonstrates the context-engineering principle: retrieve first, then construct the model input from the relevant results.

8. Comparing the Two Approaches

The transcript includes a small demonstration that compares sending a large documentation dump with retrieving a few relevant chunks.

The following figures are illustrative results from that demonstration, not universal performance guarantees.

Measure	Large context approach	Retrieved context approach
Approximate context tokens	More than 4,000	About 104
Example response time	About 2.27 seconds	About 1.72 seconds
Input-token usage	More than 4,000	About 144, including additional prompt content
Cost	Higher in the demonstration	Much lower in the demonstration
Context selection	Entire documentation dump	Three relevant chunks

The retrieved-context example used fewer tokens and had a shorter measured response time in that run. The exact savings will vary with the model, API, retrieval quality, prompt size, network conditions, and workload.

The comparison illustrates why context selection matters:

BAD APPROACH                         BETTER APPROACH

All documents                        All documents
     |                                     |
     v                                     v
Send everything                     Retrieve relevant chunks
     |                                     |
     v                                     v
Large context                        Compact context
     |                                     |
     v                                     v
More input tokens                    Fewer input tokens
     |                                     |
     v                                     v
Potentially higher cost              Potentially lower cost
and more distraction                 and more focused answers

A smaller context is not automatically better. If retrieval misses the relevant passage, the model may produce an incomplete answer. Context engineering therefore needs both efficient selection and reliable retrieval.

9. How to Evaluate Context Quality

Context engineering should be treated as an engineering discipline, not a collection of prompt-writing tricks. That means we should measure whether the context supplied to the model actually helps it complete the task.

Useful measurements include:

9.1 Retrieval relevance

Are the retrieved passages relevant to the user’s question?

A retrieval system that returns many unrelated documents is increasing context size without adding much value.

9.2 Context completeness

Does the selected context contain the facts needed to answer correctly?

A short context that omits an important exception may be worse than a longer context that includes it.

9.3 Answer quality

Does the model’s response correctly use the supplied information? Evaluate factual correctness, completeness, instruction following, and whether the answer is supported by the retrieved sources.

9.4 Token usage and cost

Measure input and output tokens, along with the cost per request. Compare these measurements across representative tasks rather than relying on one example.

9.5 Latency

Measure end-to-end response time. Retrieval, reranking, tool calls, context assembly, and model generation can all contribute to latency.

9.6 Context utilization

Check whether the model actually uses the information that was retrieved. A system may retrieve the correct passage but still fail to make it clear or prominent in the assembled context.

A practical evaluation should include both successful and difficult cases: ambiguous queries, missing information, conflicting sources, long conversations, and questions that require live tool results.

10. Common Context-Engineering Mistakes

Several mistakes can make an otherwise capable LLM application unreliable.

10.1 Sending all available documents

More context can increase cost and introduce irrelevant information. Retrieve the material that supports the current task.

10.2 Treating all conversation history as equally important

Older messages may no longer be relevant. Preserve important decisions and active constraints, and summarize or omit unrelated discussion.

10.3 Storing everything as memory

Persistent memory should contain information that is likely to be useful again and appropriate to retain. A system should not treat every message as a durable fact.

10.4 Providing too many tools

Unnecessary tool descriptions can consume context and make tool selection harder. Expose the capabilities relevant to the current task.

10.5 Compressing away important qualifications

A summary that removes dates, exceptions, units, conditions, or source attribution can change the meaning of the original information. Compression should preserve the details that affect the answer.

10.6 Ignoring source conflicts

Two documents may disagree, or an old session summary may conflict with a current system record. Define source priorities and make uncertainty explicit.

10.7 Assuming fewer tokens always means a better answer

Reducing context is useful only when the remaining information is sufficient. The objective is not the smallest possible prompt; it is the smallest context that still supports a reliable answer.

11. A Practical Context-Engineering Workflow

When designing an LLM application, it helps to approach context construction systematically.

Step 1: Understand the task. Identify what the user is asking and what information is necessary to complete it.

Step 2: Identify possible sources. Determine whether the required information is already in the current request, conversation history, a memory store, a document collection, or an external system.

Step 3: Retrieve selectively. Search only the sources that are relevant. Use suitable retrieval techniques and filter results by metadata or permissions where appropriate.

Step 4: Select tools deliberately. Make only the necessary tools available for the current task, and apply appropriate authorization and validation.

Step 5: Assemble the context. Combine instructions, the current request, relevant history, retrieved information, and tool results in a clear structure.

Step 6: Compress carefully. Remove repetition and irrelevant details while preserving facts, conditions, and source references that affect the answer.

Step 7: Generate and evaluate. Ask the model to complete the task, then evaluate the response against the source material and the expected outcome.

Step 8: Improve the system. Use failures and measurements to improve retrieval, memory selection, tool choice, context formatting, and compression.

Understand the task
        |
        v
Identify information sources
        |
        v
Retrieve relevant information
        |
        v
Select tools and memory
        |
        v
Assemble structured context
        |
        v
Compress without losing meaning
        |
        v
Send context to the LLM
        |
        v
Evaluate the result
        |
        v
Improve the context pipeline

This workflow turns context construction into a repeatable application process rather than a series of random prompt changes.

12. Key Takeaways

Context engineering expands the focus from writing a good prompt to designing the information system around an LLM.

The most important ideas are:

• Prompt engineering and context engineering complement each other. Instructions define how the model should behave; context engineering determines which information and tools it should have available.
• The context window is a limited working space. It should contain useful information for the current step.
• Relevance matters more than volume. Sending every available document can waste tokens and introduce distractions.
• RAG, memory, selective tools, assembly, and compression are important techniques for constructing useful context.
• Context is dynamic. The information needed for one step may be different from what is needed for the next.
• Measure the results. Evaluate answer quality, retrieval relevance, token usage, cost, and latency rather than assuming that a shorter prompt is always better.

The central principle is simple:

Do not just ask an LLM a question. Build a system that gives it the information and capabilities it needs to answer that question well.

That is the foundation of context engineering.