Once you've got a local AI model running, the next question is usually how to make it useful for your own stuff: your notes, your manuals, your code or your writing style. There are two main ways to do it. Retrieval-augmented generation (RAG) feeds the model relevant snippets of your documents at the moment you ask a question. Fine-tuning changes the model itself by training it on your examples. This explainer is based on documentation from IBM, AnythingLLM and Unsloth.
The short answer
| RAG | Fine-tuning |
|---|
What it changes | What the model sees for each question | The model's weights |
Best for | Answering questions from your documents | Style, format, tone and specialised tasks |
Updating your data | Add or remove documents any time | Retrain the model |
Hardware needed | Enough to run the model, plus a small embedding model | More GPU memory and time to train |
Difficulty | Easy with apps like AnythingLLM or Open WebUI | Harder: you need a training dataset |
How RAG works
IBM describes RAG as connecting a model to your own data, while fine-tuning retrains a model on a focused set of data. In a local RAG app, the process goes like this, as AnythingLLM's documentation explains:
Your documents are converted to text and split into chunks.
An embedding model turns each chunk into a list of numbers (a vector), which is stored in a vector database.
When you ask a question, your question is turned into a vector too, and the database finds the closest-matching chunks.
Those chunks are added to your prompt, and the model answers using them.
The important point, which AnythingLLM stresses, is that the model never "reads" your whole document set. It only sees the handful of chunks retrieved for each question; AnythingLLM says it gets back 4 to 6 text chunks deemed relevant to your prompt. That's why RAG is good at "what does the manual say about X?" and poor at questions that need every document at once, such as "list the three highest-calorie recipes in this folder".
You can set up RAG on a home PC in an afternoon. Our guide to chatting with your documents in AnythingLLM walks through it, and our guide to embedding models in Ollama explains the embedding step.
How fine-tuning works
Fine-tuning trains an existing model further on your own examples, so the changes live in the model itself. Unsloth's guide says fine-tuning can customise a model's behaviour, add knowledge and improve performance on specific tasks. Most home fine-tuning uses LoRA or QLoRA, which, as Unsloth explains, trains small extra matrices instead of changing every weight in the model, so only about 1% of the weights are optimised.
That makes it possible on consumer hardware. Unsloth says you can fine-tune locally with as little as 3GB of graphics memory using its tools. That figure is for small models; bigger models need more memory. The harder part is the data: you need a dataset of good examples in the format you want the model to learn. Unsloth suggests beginners start with a small instruct model and experiment from there. Our guide to fine-tuning an LLM locally with Unsloth covers the process.
Unsloth also argues that it's a misconception that fine-tuning can't teach new knowledge. That's a fair point, but for documents that change often, retraining every time is still more work than updating a RAG library.
What about just changing the prompt?
Before either, try a system prompt. Ollama's Modelfile lets you set a system prompt and parameters for a custom model in a few lines; see our guide to creating a custom model with a Modelfile. If a good prompt gets you the tone or format you want, you may not need to fine-tune at all.
Which should you use?
Questions about your documents, manuals or notes: RAG. It's quicker to set up, works with any model you can already run, and updates as soon as you add a file.
A consistent style, format or specialised task, such as always replying in a fixed structure or writing in your voice: fine-tuning, once a system prompt has proved not to be enough.
Both: they aren't exclusive. A fine-tuned model can still use RAG to look things up.
Our verdict
For most people running AI on a home PC, start with RAG. It answers questions from your own documents, needs no training, runs on the hardware you already use for local models, and you can add or remove files whenever you like. Fine-tuning is the right tool when you need the model itself to behave differently, such as a fixed style or a specialised task, and a system prompt isn't enough. It takes more effort, mainly in building a good dataset.