Context, RAG, Tools and Agents: Where Each Fails and How to Check

I wrote down three ways to compensate for an LLM's limits — context, RAG, and tools and agents — along with what I missed on the check questions. This post covers what each one is and where it fails. It does not cover implementation code or any particular framework.

1. A model only knows what you show it

A model's answer comes from what it learned plus what is in its context right now. Internal data, today's logs and your requirements were never in training, so if you do not show them, they do not exist.

A good instruction usually carries:

• Goal — what you are doing and why
• Input material — data and documents, and where they came from
• Constraints — format, scope, what not to do. "Timestamps are in KST", "mark guesses as guesses"
• Output format — a table, JSON. A fixed format also makes the result easier to check
• Examples (few-shot) — one to three input/output examples beat a long description
• What to do when it does not know — "if it is not in the material, answer 'cannot confirm'"

Giving the model a role ("you are a process engineer") helps with tone and perspective, but it does not give the model knowledge it does not have.

More context is not better. Irrelevant material makes the model miss what matters and costs more. Choosing what goes in is a skill of its own, sometimes called context engineering.

2. RAG — find it, then show it

RAG (Retrieval-Augmented Generation) means that when a question comes in, relevant documents are retrieved first and attached to the context before the model answers.

1. Chunking — split documents into pieces
2. Embedding — turn each piece into a vector that carries its meaning. Similar meaning, nearby vectors
3. Storage — put them in a vector database
4. Retrieval — turn the question into a vector too and fetch the top k nearest pieces. Often mixed with keyword search (hybrid search)
5. Generation — hand them to the model with "answer from the material below"

For a chatbot that answers from 3,000 pages of manuals that get revised every quarter, the first approach is RAG, not fine-tuning. They solve different problems.

Where RAG goes wrong

The setup: a RAG chatbot answered "the PM interval is 500 hours" and cited section 4.2 of the manual, but the current standard is 400. I gave two causes — the documents were not updated, and bad chunking pulled the wrong piece. Both were right, and I missed one: retrieval worked, and the model ignored the source and made the number up.

RAG fails in roughly four places:

• It does not fetch the document it needs — bad chunk boundaries, or terms like internal acronyms and equipment codes that embeddings do not capture
• It fetches the wrong document — and the model answers convincingly from it
• The model ignores the source and answers from its own knowledge
• The document itself is outdated or wrong — in practice, old and new editions are often indexed together and the old one gets retrieved

So the first move is to open section 4.2 yourself. If it really says 500, the problem is the documents; if it does not, the model made it up. That one check tells you which cause it is. RAG reduces hallucination; it does not remove it.

These failure points line up with the axes of RAG evaluation frameworks. The RAGAS paper looks at three things — whether the answer is grounded in the given context (faithfulness), whether it addresses the question (answer relevance), and whether the retrieved context is focused on the question (context relevance) (Es et al. 2023). The library's documentation now adds metrics such as context recall, which checks whether all the needed context was retrieved, and context precision, which checks how much of what was retrieved is useful (Ragas metrics list).

3. Tool use — have it actually run things

What the model cannot do itself, it does by calling tools.

1. The developer provides a list of available tools — name, description, input format. For example query_sensor_log(equipment_id, start, end)
2. The model outputs a request: "call this tool with these arguments"
3. The call is executed by the surrounding program, not by the model
4. The result goes back to the model
5. The model answers, or calls the next tool

Calculation done by the model itself can be wrong; done in code it is exact, and the run leaves a record you can point to. Querying live data also gets around the knowledge cutoff.

MCP (Model Context Protocol) is a standard for connecting tools and data sources to AI applications in one consistent way. Anthropic open-sourced it on November 25, 2024 (Introducing the Model Context Protocol).

4. Agents — have it loop on its own

An agent takes a goal and repeats "think, run a tool, observe the result, decide the next step" by itself. The pattern is called ReAct (Reason + Act) (Yao et al. 2022).

Given "find out why yield dropped yesterday," it roughly queries yield data to find when the drop started, finds a temperature anomaly in one chamber in the equipment logs for that window, compares only the wafers that went through that chamber, and writes a report.

The risks come from the same structure:

• Errors compound. Even at 90% per step, ten steps succeed together only about 35% of the time (0.9¹⁰). Anthropic also notes that an agent's autonomy brings higher costs and the potential for compounding errors (Building effective agents).
• It keeps going on a wrong assumption. Get the date range wrong in step one and everything after is off — while the conclusion still reads very convincingly.
• Permissions. Irreversible actions like deletion or configuration changes go through human approval, and permissions stay minimal.
• It loops without finishing. Cap the number of iterations and the cost.

When the report's conclusion looks convincing

Asked what I would check before trusting a ten-step report that concluded "a temperature sensor anomaly is the cause," I answered only "run a verification agent." Not wrong, but weak. When AI checks AI, they can miss the same errors, so the verifying agent can read the same report and find it just as convincing.

Strong verification rests on independent evidence:

1. Check the intermediate steps — the date range and time zone queried, the right line and equipment
2. Reproduce the key numbers — recompute the drop and the temperatures in code
3. Separate correlation from causation — an anomaly at the same time is not a cause. Did only the wafers through that chamber drop, are the others normal, and did anything else change in that window (a recipe change, a new material lot)?
4. Suspect the sensor itself — is the reading a real process problem or a failing sensor?

Asked again, I said I would tell it to "explore the intermediate steps, verify the numbers with code, and separate correlation from causation." Two more things make that stronger:

• Give it the raw data and run logs, not just the report. With only the report, it can only check the report's internal logic.
• Give it the role of refuting. "Find evidence that this conclusion is wrong. Propose at least two other candidate causes and rule each out with data." Take the output as pass/fail per item, with evidence.

Wrapping up

• A model can only use what you put in its context, so it helps to include only what is needed.
• RAG attaches retrieved documents, but it can fail through missed retrieval, wrong documents, ignored sources or outdated documents, so open the cited source yourself.
• With tools, calculation and lookups run as code and leave a record.
• Errors pile up over longer agent runs, so check the intermediate steps and key numbers independently.