Fine-tuning an LLM: when it’s worth it, and when it isn’t
Fine-tuning gets asked about a lot, usually in the form “should we train a model on our data?” The short answer, most of the time, is not yet. Here’s what fine-tuning actually is, what it’s good at, and how to tell whether your problem needs it.
What fine-tuning actually does
A large language model arrives pre-trained on a vast amount of general text. Fine-tuning continues that training on your own examples, so the model’s internal weights shift towards your task. That’s the key difference from the other two ways of adapting a model. Prompt engineering changes the instructions you send but leaves the model untouched. Retrieval-augmented generation (RAG) fetches relevant documents at the moment of the request and pastes them into the context, so the model can read them without ever being retrained.
Only fine-tuning changes the model itself. That’s its power and its cost.
What it’s good at
Fine-tuning teaches behaviour, not knowledge. It’s the right tool when you need a model to reliably produce a specific format, follow a house style, classify things your way, or handle a narrow task the base model does clumsily. Given a few hundred good examples of input and desired output, a fine-tuned model will often do in one short prompt what previously took a page of instructions and still came out inconsistent.
What it does badly is learn facts. If you want a model to know your product catalogue, your policies, or anything that changes weekly, fine-tuning is the wrong tool. The model won’t reliably recall specifics from training data, and every update means retraining. RAG handles that problem better and cheaper, because the facts live in your documents and get retrieved fresh each time.
The ladder worth climbing in order
A sensible order of escalation: try prompting first, add retrieval if the model needs your information, and fine-tune only when the first two rungs genuinely fall short. Each step up costs more to build and more to maintain. A surprising share of “we need to fine-tune” conversations end at rung one, because a well-written prompt with a few examples in it gets close enough.
If you do fine-tune
You don’t need to retrain a whole model. Parameter-efficient methods, LoRA being the common one, train a small adapter on top of a frozen base model. QLoRA pushes this further by quantising the base model, which brings a 7 or 8 billion parameter open-weight model within reach of a single decent GPU. Hosted alternatives exist too: several providers let you upload examples and fine-tune their models through an API, which trades control for convenience.
Whichever route you take, two things matter more than the method. First, data quality beats data volume; a few hundred carefully checked examples outperform thousands of scraped ones. Second, build an evaluation set before you train, not after. Without a fixed set of test cases scored before and after, you cannot tell whether the fine-tune helped, hurt, or did nothing.
The honest caveats
Fine-tuned models drift out of date as base models improve, so you’re signing up for maintenance, not a one-off job. Training data can leak into outputs, so keep personal data out of it. And a model tuned hard on one task can get worse at everything else. None of these are reasons to avoid fine-tuning; they’re reasons to make sure the problem justifies it first.
Start with the cheapest thing that could work. Fine-tune when the evidence says you must.
