The request usually arrives in the form “we want to train the model on our data”. Almost always, what is wanted is for the system to answer questions using company documents — and fine-tuning is the wrong tool for that, in a way that is not obvious and is expensive to discover late.

The distinction is worth getting right before any money is spent, because the two approaches differ in cost, in how they fail, and in what happens when the underlying information changes.

What each one actually changes

Retrieval-augmented generation
The system searches your material for passages relevant to the question, puts them in front of the model, and asks the model to answer using them. The model has learned nothing new. It is being handed the source at the moment it answers, the way a person answers a question with the manual open.
Fine-tuning
The model’s weights are adjusted using examples of desired input and output. It changes how the model behaves — its format, its tone, its willingness to follow a particular structure, its handling of a specialised task. It is teaching a habit rather than a fact.

The test

Ask what is actually wrong with the output you are getting now.

It does not know something
Your policies, your product catalogue, this customer’s order history, what your company decided last quarter. This is a knowledge problem, and the answer is retrieval. Every time.
It knows, but behaves wrongly
It will not stick to the output format however clearly you ask. It writes in the wrong register. It will not reliably follow a specialised classification scheme with forty categories. This is a behaviour problem, and it is the case where fine-tuning is genuinely the answer.

Why retrieval is the right default

For nearly every business use case, retrieval wins on grounds that have nothing to do with quality of output.

  • The answer changes when the document changes. Update the source and the next answer is correct — no retraining, no redeployment, no version of the model carrying last quarter’s policy around.
  • It can cite. You can show which passage an answer came from, which is what makes the answer checkable and what makes the system trustable by the people who have to use it.
  • It can decline. If retrieval returns nothing relevant, the agent has explicit grounds to say it does not know, and an agent permitted to decline is far more useful than one that is not.
  • Access control survives. You can filter what a given user is allowed to retrieve. A fine-tuned model has no equivalent — whatever went into training is available to everyone who can reach the model.
  • It is cheaper to be wrong. A bad retrieval configuration is a configuration change. A bad training run is a training run.

That fourth point is the one that ends the discussion in most regulated environments, and it is routinely missed at the design stage. If some of your documents should not be visible to everyone, fine-tuning on them is not an option — it is a data exposure with extra steps.

When fine-tuning genuinely earns its place

It is not a bad technique. It is a specific one, and there are cases where nothing else does the job:

  • A rigid output format the model keeps drifting away from, at volume, where the drift is a real cost.
  • A classification task with many categories and subtle boundaries, where you have thousands of correctly labelled examples and the labels encode judgement no instruction can express concisely.
  • A specialised register — a domain that writes in a way general text does not, where matching the convention is part of being correct.
  • Cost or latency at scale: a smaller fine-tuned model matching a larger general one on a narrow task can be the right economic trade, once the volume justifies the work.

Notice that every one of those is about behaviour, and that all of them assume you already have the data to demonstrate the behaviour you want. If you do not have several hundred clean examples of correct handling, fine-tuning is not available to you yet regardless of whether it is appropriate.

What teams get wrong about retrieval

Choosing retrieval is the easy part. Most disappointing retrieval systems are disappointing for one of three reasons, none of which is the model.

The source material was never usable
Retrieval needs something to retrieve from. A well-maintained help centre is close to free to work with. A shared drive of PDFs — some scanned, some superseded, none labelled with which is current — is a real project before any agent work starts, and it is the most commonly underestimated line in an automation budget.
Retrieval quality was never measured
The failure is almost always that the right passage was not returned, not that the model mishandled it. That is measurable separately from the answer, and measuring it separately is what turns debugging from guesswork into work. Teams that only evaluate final answers spend months tuning prompts to fix a search problem.
Chunking was treated as a detail
How documents are split determines what can be found. Split too small and a passage loses the context that made it meaningful; too large and the relevant sentence is buried among irrelevant ones. There is no universally correct answer, which means it has to be tested against real questions rather than chosen once.

All three of these are the same lesson as the wider one about agents in production: the model is rarely the component that failed, and the failure is usually in the plumbing around it.

The order to do this in

  1. Write down what is wrong with the current output, in terms of specific cases. “It does not know our returns policy” and “it will not stay in the required format” are different projects.
  2. If anything on that list is a knowledge gap, build retrieval first. It will resolve more of the list than expected, including things that looked like behaviour problems.
  3. Measure what is left. Keep the cases retrieval did not fix, with their correct handling written down. This set is the only honest basis for deciding what to do next.
  4. Only then consider fine-tuning, and only for the behaviour that survived. You now have both a reason and the training examples, which is the first point at which the option is real.

In practice most projects stop after step two, and that is the correct outcome rather than a shortcut. Retrieval against material you control, with an explicit path for “I do not have this”, covers the overwhelming majority of what businesses actually want from an agent.

Retrieval changes what the system knows. Fine-tuning changes how it behaves. Almost everyone asking for the second one wants the first.