Running a local language model is now easy. Fine-tuning one, so that it answers in your format, follows your rules or knows your specialist subject, used to need data-centre hardware. Unsloth, a free open-source fine-tuning toolkit, has changed that: with a method called QLoRA, small and medium models can be fine-tuned on a gaming graphics card.
Forge PC hasn't run fine-tuning benchmarks itself. Everything below, including the VRAM figures, comes from Unsloth's official documentation.
Do you actually need to fine-tune?
Fine-tuning changes a model's weights. Unsloth's guide gives examples such as training a model to judge whether a headline is good or bad news for a company, or to answer in the style of past customer conversations. If you just want a model to answer questions about your documents, retrieval tools may be simpler; see how to chat with documents in AnythingLLM. If you only want a different system prompt or settings, an Ollama Modelfile is far quicker.
What hardware you need
Unsloth lists these requirements for training on Windows:
Windows 10 or Windows 11 (64-bit)
An NVIDIA GPU with drivers installed. Unsloth says it supports NVIDIA GPUs from 2018 onwards with a minimum CUDA capability of 7.0, which includes RTX 20-series cards and newer. GTX 1070 and 1080 cards work but are slow.
Git, and Python 3.11, 3.12 or 3.13, used inside a Python environment
Unsloth Studio can train on NVIDIA, Apple Silicon (MLX) and Intel devices, and the code-based version can train on AMD and Intel GPUs. It also works without a GPU for running models, but training is where a graphics card matters.
How much VRAM fine-tuning needs
Unsloth publishes minimum VRAM figures by model size. QLoRA loads the base model in 4-bit; LoRA uses 16-bit. Unsloth stresses these are absolute minimums and some models need more.
Model size | QLoRA (4-bit) | LoRA (16-bit) |
|---|
3B | 3.5GB | 8GB |
7B | 5GB | 19GB |
8B | 6GB | 22GB |
14B | 8.5GB | 33GB |
27B | 22GB | 64GB |
32B | 26GB | 76GB |
70B | 41GB | 164GB |
On those figures, an 8GB card can QLoRA fine-tune models up to about 11B parameters (7.5GB), a 12GB card reaches the 14B class, a 24GB card the 27B class, and 16-bit LoRA on anything above 3B needs workstation-class memory. If you run out of memory, Unsloth's first suggestion is to lower the batch size to 1, 2 or 3. For buying advice, see our best GPU for local LLMs.
LoRA or QLoRA?
Unsloth explains that both methods leave most of the original model untouched and train small extra "adapter" matrices, so only around 1% of the weights are optimised. QLoRA also quantises the base model to 4-bit to save about 75% of the memory. Unsloth recommends beginners start with QLoRA and a small instruct model (it suggests an 8B model), and says its dynamic 4-bit quants have largely recovered the accuracy gap between QLoRA and LoRA. It also advises against jumping straight to full fine-tuning: if an idea doesn't work with LoRA or QLoRA, it almost certainly won't work with full fine-tuning either.
Step 1: Install Unsloth on Windows
Unsloth offers three routes:
Unsloth Desktop, a native app: download it, launch it and wait for setup to finish. This is the easiest option.
Unsloth Studio, a browser-based interface for fine-tuning without writing code. Unsloth says it runs directly on Windows without WSL.
Unsloth Core, the original code-based library, installed with Conda, Docker or WSL. Unsloth notes that if you plan to use vLLM (for example for reinforcement learning), it only works on Windows through WSL or Linux; see our guide to setting up WSL2 for local AI.
Update your NVIDIA driver before you start.
Step 2: Choose a model and method
In Unsloth Studio, pick a base model from Hugging Face or your local files, then choose QLoRA, LoRA or full fine-tuning. Unsloth fills in sensible default settings for the model you choose. Note that GGUF files can't be used for training; they're for running models only. Gated models such as Llama and Gemma need a Hugging Face access token.
Step 3: Prepare your dataset
This is the part that decides how good the result is. Unsloth recommends a well-structured dataset, ideally question-and-answer pairs, and says simply dumping in raw documents won't work as well. Studio can search Hugging Face datasets or take your own uploaded files, and will try to detect the format and map columns automatically.
Step 4: Train and test
Start training and watch the live progress, GPU statistics and charts in Studio. Unsloth recommends a context length of 2048 for test runs. Keep first runs short, check the results, then adjust your data rather than just training for longer.
Step 5: Export and run it
When you're happy, use Studio's Export option to save the model as GGUF, Safetensors or a LoRA adapter. Unsloth says these can be used in llama.cpp, Ollama, vLLM and other tools. A GGUF export can then be loaded in Ollama or LM Studio like any other model; see how to run AI models locally with Ollama and LM Studio and GGUF quantisation explained.
Our verdict
Unsloth makes fine-tuning possible on a home PC: on Windows you need an NVIDIA GPU with CUDA capability 7.0 or newer (RTX 20-series onwards), and with QLoRA Unsloth's minimums are about 6GB of VRAM for an 8B model, 8.5GB for 14B and 22GB for 27B. Start with Unsloth Desktop or Studio, a small instruct model and a clean question-and-answer dataset, then export to GGUF to run the result in Ollama or LM Studio. Forge PC hasn't run fine-tuning benchmarks itself.