RAG Architecture: When to Use Retrieval Augmented Generation

An LLM's knowledge ends at its training cutoff and does not include your documents, your pricing, your support history or your internal processes. Asking the model directly means hallucination risk on exactly the questions where accuracy matters most.

The common reaction is fine-tuning — but fine-tuning does not add knowledge, it changes behavior. A model trained on your documents can still confidently invent a policy that does not exist. Retrieval, not memorization, is how you give a model access to facts.

Fine-tuning teaches style, not facts. If the answer must be true, the facts must be retrieved.

• Does the answer depend on private or changing data? — yes means RAG.
• Is the problem about format, tone or domain behavior? — that is fine-tuning territory.
• Does the user need sources and citations? — RAG gives you a natural trace.
• Can the data change monthly or weekly? — RAG updates in an hour; retraining does not.

Most production systems combine both: a base model chosen for reasoning, RAG for grounding, and light fine-tuning or prompt work for tone — in that order of investment.

RAG is three systems, not one script:

• Ingestion — documents are chunked, embedded and stored. Chunking strategy is a product decision: a legal contract chunks differently from a knowledge base article.
• Retrieval — the user question is embedded, then searched. We combine semantic search with keyword and metadata filters, because semantic alone misses exact terms and IDs.
• Synthesis — retrieved chunks go to the LLM with strict instructions: answer only from the context, cite the source, say “not found” instead of guessing.

The retrieval step is where quality is won or lost. If the right chunk does not get retrieved, no model can produce the right answer.

Chunking is a product decision, not a text-processing detail. The same document type chunks differently depending on how users will query it:

• Legal contract — split by clause hierarchy: sections and clauses keep their boundaries, so clause-level queries return the exact paragraph, not a mixed fragment.
• Knowledge base article — split by headings and paragraphs with semantic boundaries, so each chunk answers one topic and stays self-contained.
• Support history — split by conversation turns with timestamps, so retrieval can prefer recent answers over outdated ones.

Every chunk carries metadata. Metadata is what turns retrieval from a keyword match into a precise lookup:

{
  "document_id": "contract-2026-041",
  "document_type": "contract",
  "source_system": "legal_repository",
  "language": "en",
  "clause": "termination",
  "clause_number": "12.3",
  "chunk_index": 4,
  "updated_at": "2026-07-30"
}

Filters on these fields are part of the retrieval query: only the tenant's documents, only the right language, only documents updated after a date. Metadata is also how access control is enforced at retrieval time.

• Chunking by structure — split on semantic boundaries (headings, paragraphs, sections), not fixed character counts.
• Hybrid search — vector similarity plus keyword and metadata filters; re-ranking the top candidates by relevance.
• Embeddings — a good general embedding model is enough for most business data; re-embedding on content updates keeps freshness.
• PostgreSQL with pgvector — one database for state, logs and vectors simplifies operations and keeps data in one place.
• Evaluation set — a fixed set of questions with expected chunks and answers. Every change to chunking or prompts runs against it.
• Fallback behavior — the system is trained to refuse without context: “no answer in the knowledge base” is a valid, honest answer.

• Fixed-size chunking — cutting documents by character count destroys structure: clauses, headings and questions land in different chunks and retrieval misses the connection.
• Semantic-only search — vector similarity alone misses exact terms, codes and IDs. Without keyword search and metadata filters, precise queries fail silently.
• No re-ranking — top-k by similarity returns the most similar, not the most relevant. Re-ranking the candidates fixes the difference.
• No metadata filters — mixing languages, document types and dates in one index pollutes results. Filters are cheap and change retrieval quality more than any model choice.
• No evaluation set — chunking, embeddings and prompts are changed without a regression test. Retrieval quality drifts invisibly with every change.
• Answering without context — letting the model answer when nothing was retrieved. The refusal path is part of the architecture, not an error state.
• Stale index — documents change, but the index does not. Ingestion as a pipeline, not a one-time job, is what keeps answers current.

We use the same pipeline design in our own products:

• AI Business Copilot — a RAG knowledge base stores company context, and multi-agent analysis retrieves from it when searching for automation opportunities.
• LegalFlow AI — document processing for legal teams: contracts and cases are chunked by structure, retrieved by clause-level queries and answered with source references.
• LeadForge AI — a retrieval layer feeds the content agent with company data and CRM context so generated outreach is grounded in facts, not generic templates.

The pattern repeats across very different domains: grounded answers, source traceability and honest refusal when the knowledge base has nothing.