Grounding with Retrieval

Roadmaps › AI Engineering

Scenario

You ask an internal support bot: 'What is our SLA for enterprise customers?' It confidently answers '99.9%'. The real answer is '99.99%'. Before you read on, predict: did the model lie, or is something else happening?

Think about where the model's answer came from — then read on to understand hallucination and how retrieval fixes it.

Mental model

RAG turns a closed-book exam into an open-book exam. Instead of guessing from memory, the model reads the answer off the page you hand it.

Standard LLMs answer from weights learned during training. When they do not know the answer, they do not say 'I don't know' — they generate the most likely-sounding next token, which is often wrong. Grounding fixes this: you retrieve the relevant document text and paste it directly into the prompt. The model's job shifts from 'remember the answer' to 'read the answer out of the provided text'.

Deep dive

LLMs are excellent at reading, summarizing, and reasoning. They are terrible at storing facts. Their training data has a cutoff date, and they have no knowledge of your private company documents. If you ask a niche or private question, they will confidently hallucinate an answer that sounds right but is wrong.

Models reason well but store facts poorly. Questions about private or recent data produce hallucinations.

The fix is to stop relying on the model's memory. Use a search system to find the relevant information, then paste that text directly into the prompt. The model reads the answer instead of guessing. The model's reasoning ability is the same — but now it has the right facts in front of it.

Provide the facts in the prompt so the model reads instead of guessing.

This workflow is called Retrieval-Augmented Generation (RAG). It has two steps: retrieve the right facts from a database, then generate an answer using only those facts. The pipeline is: User query → Vector search → Retrieve relevant chunks → Inject into prompt → Model reads and replies.

RAG pipeline: Retrieve the facts, Augment the prompt, Generate the answer.

The prompt must include strict instructions to anchor the model. You tell it explicitly: 'Answer ONLY using the provided context. If the answer is not in the context, say I don't know.' Without this instruction, the model may combine the retrieved text with hallucinated facts. This explicit instruction is the 'grounding'.

Grounding = context in the prompt + strict instruction to read only from that context.

RAG shifts the burden of accuracy from the model to your search system. If the retriever finds the wrong paragraphs, the model gives the wrong answer — or correctly says 'I don't know'. If it finds the right paragraphs, the model almost always succeeds. AI engineers spend most of their time improving retrieval, not the generation step.

The model can only use what the retriever finds. If retrieval fails, the bot fails.

Code examples

Without grounding — hallucination risk

from openai import OpenAI
client = OpenAI()

# Model guesses from training data — private facts will be hallucinated
response = client.chat.completions.create(
    model='gpt-4o',
    messages=[{'role': 'user', 'content': 'What is the SLA for RetainHQ Enterprise?'}]
)
print(response.choices[0].message.content)  # Likely wrong

Without context, the model relies on pre-trained weights. For private company data it was never trained on, it will generate a plausible-sounding but incorrect answer.

With grounding — model reads the answer instead of guessing

from openai import OpenAI
client = OpenAI()

# 1. Retrieve the relevant fact (from your vector store, simplified here)
retrieved_docs = 'RetainHQ Enterprise accounts guarantee a 99.99% uptime SLA.'

# 2. Augment the prompt with the context and strict instructions
grounded_prompt = f'''
Answer the question using ONLY the provided context.
If the context does not contain the answer, say "I don't know."

Context:
{retrieved_docs}

Question: What is the SLA for RetainHQ Enterprise?
'''

# 3. Generate the grounded answer
response = client.chat.completions.create(
    model='gpt-4o',
    messages=[{'role': 'user', 'content': grounded_prompt}]
)
print(response.choices[0].message.content)  # 99.99%

The model reads the exact fact from the retrieved context. The strict instruction tells it to refuse if the answer is not present — this prevents the model from falling back to hallucination.

Key points

Common mistakes

Glossary

Retrieval-Augmented Generation
A pattern where you search your private documents for relevant facts and paste them into the prompt so the AI reads them instead of guessing from memory.
hallucinate
When an AI does not know the answer but invents a confident, wrong response instead of saying 'I don't know'.
grounding
Giving the model explicit text to read and strict instructions to answer only from that text, so it cannot invent facts.

Recall questions

Questions & answers

A team's RAG chatbot still hallucinates on questions about features that do not exist. How would you debug and fix this?

The model is falling back to its internal knowledge when the retriever returns nothing. Two fixes: first, add explicit grounding instructions ('Answer ONLY from the provided context, otherwise say I don't know'). Second, add a minimum similarity threshold so that a low-quality retrieval returns zero chunks — which forces the model into its 'I don't know' fallback state.

Approach: Identify the model falling back to internal knowledge as the root cause. The two fixes are grounding instructions and a similarity threshold to trigger the refusal path explicitly.

You have a large corpus of private documents and want to build a Q&A bot. Should you fine-tune the model on the documents or use RAG?

Use RAG. Fine-tuning is for teaching a model a new style, format, or reasoning pattern — it is poor at memorizing new facts and still hallucinates them confidently. RAG is cheaper, lets you update or delete documents without retraining, provides citations, and prevents hallucination through grounding.

Approach: Contrast fact retrieval (RAG) vs behavior modification (fine-tuning). Fine-tuning does not reliably store new facts. RAG is the correct tool for document Q&A.

Continue learning

Previous: Embeddings

Next: Chunking Strategies

Next: Hallucination Mitigation

Return to AI Engineering Roadmap