RAG vs. Fine-Tuning: How to Choose the Right Technique for Enterprise LLM Systems

RAG and fine-tuning are often presented as competing answers to one question: should we give company documents to the model, or train the model on company data? That framing is incomplete. RAG changes which knowledge is available at answer time. Fine-tuning changes how a model behaves on a task. A sound production system chooses where rules, RAG, fine-tuning, and human review are actually needed.

This is TankDev's decision framework: retrieve the source first when the work requires a current contract, price, procedure, or customer record; evaluate fine-tuning experimentally when the problem is a repeatable response format, classification boundary, or behaviour; and make deterministic services—not an LLM—the decision maker for exact calculations, authorization, and irreversible actions.

1. Separate the problem first: knowledge, behaviour, or rule

When a user asks, “What is the return window for this customer?” the problem is access to current knowledge. The policy may change tomorrow, and the evidence for a correct answer is the relevant policy revision. That is a RAG problem. “Classify an incoming email into one of six operations categories and return valid JSON only” is a consistency and output-contract problem; it may be a fine-tuning candidate. “A second approval is required above a 5,000 TRY refund” is a deterministic business rule and belongs in SQL, a workflow, or a rules service.

2. What RAG is—and is not

Retrieval-augmented generation selects evidence from authorized sources for a user's question and supplies that evidence to the model context. A production path acquires sources, prepares text and metadata, retrieves candidates, reranks evidence, produces a cited answer, and records observability events. A vector database can be one component of this path; it is not a synonym for RAG.

Good RAG does not end at splitting a PDF into smaller pieces. It preserves revision, effective date, product scope, language, access control, heading hierarchy, and source link. Retrieval may combine lexical and semantic search, then use a reranker to narrow evidence. The model should answer from evidence or explicitly report that evidence is insufficient.

{
  "query": "What is the return window in the 2026 dealer agreement?",
  "actor_id": "dealer-42",
  "filters": {"locale": "en", "valid_on": "2026-09-27"},
  "retrieved": [{"doc_id": "returns-2026-07", "revision": 7, "chunk_id": "4.2", "score": 0.91}],
  "answer_policy": "cite_or_abstain"
}

Here, actor_id is not merely a log field: retrieval must filter out customer or contract fragments that the actor cannot access. Adding “do not reveal private information” to a prompt afterward is not access control. If retrieval returns no sufficient evidence, the system should enter an insufficient_evidence state rather than generate a plausible answer.

3. What does fine-tuning change?

Fine-tuning moves a base model toward selected input-output examples for a specific task. Classification labels, structured output format, domain language, concise response style, or preferred tool selection can be suitable targets. A provider treats it as a separate operation with a training dataset and job; for example, the OpenAI API specifies JSONL training data for a fine-tuning job. OpenAI fine-tuning reference

Fine-tuning does not turn training facts into a reliable document store. When a policy or price changes, it is hard to identify every affected answer. A memorized-looking example is neither citable, current, nor access-controlled evidence. Knowledge-base questions can therefore require RAG alongside a fine-tuned model.

• A training example represents the ideal work outcome and permitted output contract.
• A validation set supports prompt, hyperparameter, or decision-threshold choices.
• A test set stays untouched until final comparison; the same customer, template, or document family must not leak across splits.
• Model, dataset, prompt, and evaluation versions belong in the same release record.

4. Concrete example: a dealer support assistant

A B2B support assistant can perform three different jobs. When a user asks for the current return condition, the system retrieves the authorized contract revision with RAG and displays the clause link. When a user submits free-form text, a model classifies it as returns, pricing, delivery, account, technical, or other. Before an action creates a return, a deterministic service validates the customer, order, time window, and authorization.

For classification, compare a baseline prompt against 600 labelled examples on a held-out test set. If failures are mostly caused by changing policy knowledge, fine-tuning is the wrong investment; repair retrieval, filters, or source quality. If evidence is present but category format or domain language stays inconsistent, a fine-tuning experiment has a meaningful hypothesis. This distinction prevents the costly reflex to train first.

5. How do you measure RAG quality?

A convincing answer is not proof that the system is good. RAG evaluation needs at least two layers: did retrieval bring back the correct evidence, and did generation provide a correct and sufficient answer using only that evidence? A wrong document can be summarized fluently. A correct document can be used to support the wrong clause. These failures need different owners and different fixes.

An evaluation set should include easy FAQs alongside name collisions, stale revisions, multi-document answers, permission boundaries, OCR errors, empty retrieval, and conflicting sources. Google Cloud's RAG guidance also emphasizes that retrieval quality is central and irrelevant retrieval can lead to off-topic or incorrect generation. Google Cloud RAG guide

6. Make fine-tuning a measured experiment

Fine-tuning is not a deployment decision; it is an experiment. State a measurable objective first: “the JSON schema is valid” or “priority classification reaches at least X accuracy under expert adjudication.” Compare a base model with a strong system prompt, a base model with RAG, and a fine-tuned version using blind evaluation on the same test set. More complexity does not make an approach more correct.

Classify failure examples: wrong knowledge, wrong retrieval, schema failure, wrong tool selection, missing context, safety violation, or evaluation ambiguity. Adding every failure to training can harm dataset quality. Examples should show ideal behaviour; personal data, access keys, stale conflicting rules, and unverified model output do not belong in training data.

7. Cost and latency are different accounting models

RAG cost includes source preparation and indexing plus query-time embedding, search, reranking, context tokens, and generation. When a document changes, only affected fragments can be processed again. Fine-tuning has dataset preparation, training-job, evaluation, and custom-model inference costs. The material cost, however, includes correction effort, operational delay, and the impact of a wrong answer—not the model invoice alone.

8. Security, access, and data lifecycle

Before a RAG system adds a document to context, it applies tenant, role, record-level, contract-scope, and effective-date filters. Source fragments, answer, and actor identity may be recorded for observability with data minimization. When a document is deleted or permission is revoked, remove it from the index and caches as well.

Fine-tuning data adds a second governance question: examine the provider's data retention, usage, and deletion terms. Governance is not solved by calling a file “fine-tuning data.” Define which data class may leave the boundary, how long it is retained, and how a model revision is retired.

9. A hybrid architecture is often the right result

Mature systems can use both methods: a fine-tuned or carefully prompted model structures the query and stabilizes response form; RAG brings current, authorized evidence; a rules service validates permissions, calculations, and side effects; human review closes uncertain or high-risk cases. This assigns each component the responsibility it can actually carry instead of trying to teach everything to a model.

request → authorization → query rewrite → retrieval + ACL filter → rerank
       → evidence check → LLM response → schema validation
       → deterministic business rule → action | review queue
       → audit log + evaluation event

10. A practical decision sequence

• Write the success contract: what are the correct source, required fields, side effect, and deadline?
• Establish a simple baseline: strong prompt, schema validation, rule checks, and a measured example set.
• When knowledge changes, start with versioned sources and RAG; evaluate retrieval separately.
• When a behaviour problem repeats, test the fine-tuning hypothesis on a fixed test set.
• Do not execute a high-risk decision without deterministic validation or human approval.
• Monitor versions, cost, false acceptance, source evidence, and rollback in production.

For TankDev, the right question is not “RAG or fine-tuning?” It is which work outcome needs current evidence, which behaviour needs examples, which decision needs an exact rule, and which exception requires human responsibility. Our AI automation architecture guide explains how these components meet APIs, queues, validation, and monitoring.