A local AI model that answers in a blur on one PC can crawl on another with a similar graphics card. Usually the difference is not raw GPU power but memory: whether the whole model, plus its working memory for the conversation, fits in your card's VRAM. Once part of it spills into system RAM, speed drops sharply.
Forge PC has not benchmarked these settings, so we do not quote speed-up figures. The steps below follow Ollama's official documentation (its FAQ and context length guide). If you are new to Ollama, start with our guide to running AI models locally with Ollama or LM Studio.
1. Check whether the model fits on your GPU
Before changing anything, load your model and run:
ollama ps
Ollama's documentation explains the PROCESSOR column. 100% GPU means the model is loaded entirely into the graphics card. A split such as 48%/52% CPU/GPU means part of it is in system memory. Ollama's context length guide says that for best performance you should "avoid offloading the model to CPU" and use this command to verify the split. If you see anything other than 100% GPU, the rest of this guide is about getting there. To keep an eye on GPU memory while you work, see our HWiNFO and Afterburner monitoring guide.
2. Pick a model and quantisation that fit
The single biggest lever is the model file itself. A smaller model, or a more heavily quantised version of the same model, needs less memory. Our quantisation explainer explains the quality trade-off of formats such as Q4 and Q8, and our local AI GPU guide shows which sizes suit 8GB, 12GB, 16GB and 24GB cards.
3. Set the context length deliberately
Context length is how many tokens the model can hold in memory for a conversation. Ollama's guide says that "setting a larger context length will increase the amount of memory required to run a model", and its defaults depend on your VRAM:
Your VRAM | Ollama's default context |
|---|
Under 24GB | 4K tokens |
24GB to 48GB | 32K tokens |
48GB or more | 256K tokens |
Source: Ollama context length documentation.
That default is a sensible starting point for chat. Ollama does recommend at least 64,000 tokens for tasks like web search, agents and coding tools, but on a 12GB or 16GB card that much context can push a model off the GPU. Raise it only as far as you need, and check ollama ps afterwards.
To change it, Ollama's documentation gives several options: the context slider in the Ollama app's settings, the OLLAMA_CONTEXT_LENGTH environment variable when serving, or, inside an ollama run session, /set parameter num_ctx 8192 (replace the number with the size you want). API users can pass num_ctx in the request options.
4. Make sure Flash Attention is on
Ollama's FAQ describes Flash Attention as a feature of most modern models that "can significantly reduce memory usage as the context size grows". Ollama now uses it automatically when your backend and GPU support it. You can force it on with the environment variable OLLAMA_FLASH_ATTENTION=1 (or off with 0) if you need to.
5. Quantise the KV cache
The KV cache is the model's working memory for your conversation, and it grows with context length. Ollama can store it at lower precision when Flash Attention is enabled, using the OLLAMA_KV_CACHE_TYPE environment variable:
Setting | Memory use | Ollama's description |
|---|
f16 (default)
| Full | High precision |
q8_0
| About half of f16 | Very small loss in precision, usually no noticeable impact; recommended if not using f16 |
q4_0
| About a quarter of f16 | Small to medium loss in precision, more noticeable at higher context sizes |
Source: Ollama FAQ.
q8_0 is the safe first step. Ollama warns that this is a global setting for all models, and that models with a high GQA count, such as Qwen2, may be affected more, so test the quality on your own tasks.
6. Keep one model loaded and limit parallel requests
Ollama can load several models at once and handle parallel requests, but both cost memory. The FAQ explains that parallel requests multiply the context: a 2K context with four parallel requests becomes an 8K context with the extra memory that implies. When using the GPU, new models must fit completely in VRAM to load alongside others.
If you only chat with one model at a time, setting OLLAMA_NUM_PARALLEL=1 keeps memory use predictable. The keep_alive setting (or OLLAMA_KEEP_ALIVE) controls how long a model stays loaded after use, which avoids the delay of reloading it from disk every time.
7. Set environment variables properly on Windows
Ollama's FAQ says that on Windows it inherits your user and system environment variables. To change one:
Quit Ollama from the taskbar.
Open Settings and search for environment variables.
Choose Edit environment variables for your account.
Add or edit the variable (for example OLLAMA_KV_CACHE_TYPE with the value q8_0).
Save, then start Ollama again.
If it is still slow
If a model you need simply will not fit, more VRAM is the real fix. Our RAM for local AI guide explains when system memory helps, and our guide to running a local LLM across two GPUs covers splitting a model between cards.
Our verdict
Most slow local AI comes down to memory. Run ollama ps first: if it does not say 100% GPU, choose a smaller model or quantisation, keep the context length no bigger than you need, make sure Flash Attention is on and try OLLAMA_KV_CACHE_TYPE=q8_0, which Ollama says roughly halves KV cache memory with very little quality loss. Limiting parallel requests and keeping a model loaded also help. Forge PC has not benchmarked these settings; every step follows Ollama's own documentation.