Skip to content

How-ToRAM

LLM Context Length Explained: Why Longer Chats Need More VRAM

Context length is how many tokens a local AI model can see at once, and its KV cache uses extra memory as it grows. Ollama defaults to a 4k context on cards with under 24GiB of VRAM. Raise it deliberately, check ollama ps keeps the model 100% on the GPU, and use a q8_0 KV cache to roughly halve cache memory.

By , Editorial team

3 min read

LLM Context Length Explained: Why Longer Chats Need More VRAM

When you run a large language model locally, the model file isn't the only thing that has to fit in memory. The context length, the number of tokens the model can "see" at once, also takes memory, and it can be the reason a model that fitted your graphics card suddenly spills onto the CPU and slows down.

Forge PC hasn't benchmarked specific models for this explainer. It's based on Ollama's documentation and Hugging Face's guide to the KV cache.

What context length means

Ollama defines context length as the maximum number of tokens the model has access to in memory. Your prompt, any documents you paste in, the chat history and the model's reply all count towards it. When a conversation exceeds the context window, older material falls out and the model can no longer refer to it.

Ollama's documentation says tasks that need a large context, such as web search, agents and coding tools, should be set to at least 64,000 tokens.

Why it uses memory: the KV cache

As a model processes text, it stores intermediate results for every token, called the key-value (KV) cache, so it doesn't have to recompute them for each new word. Hugging Face's documentation explains that the KV cache can occupy a significant portion of memory and become a bottleneck for long-context generation. The longer the context, the bigger the cache.

So total memory use is roughly the model weights plus the KV cache plus some overhead. Ollama warns that setting a larger context length will increase the amount of memory required to run a model, and that you should make sure you have enough VRAM before raising it.

Ollama's defaults

Ollama's context length documentation says it picks a default based on your VRAM:

Your VRAM

Default context

Under 24GiB

4k tokens

24-48GiB

32k tokens

48GiB or more

256k tokens

That means most gaming graphics cards, with 8GB to 16GB, start at a 4k context. That is fine for short chats, but too small for long documents or coding agents.

How to change the context length in Ollama

Ollama's documentation lists several methods:

  • Ollama app: move the context length slider in Settings.

  • For the whole server: start it with the OLLAMA_CONTEXT_LENGTH environment variable, for example OLLAMA_CONTEXT_LENGTH=64000 ollama serve.

  • In a chat: type /set parameter num_ctx 8192 inside ollama run.

  • Through the API: pass "num_ctx" in the request options.

Then run ollama ps. Its PROCESSOR column shows how much of the model is on the GPU (ideally "100% GPU") and the CONTEXT column shows the allocated context. Ollama recommends avoiding offloading the model to the CPU for best performance. If raising the context pushes part of the model onto the CPU, expect generation to slow sharply.

How to fit a longer context into your VRAM

  1. Quantise the KV cache. With Flash Attention enabled, Ollama lets you set OLLAMA_KV_CACHE_TYPE. Its FAQ says q8_0 uses about half the memory of the default f16 with a very small loss in precision, and q4_0 about a quarter, with a small-to-medium loss that may be more noticeable at higher context sizes. The setting applies to all models.

  2. Check Flash Attention. Ollama's FAQ says Flash Attention can significantly reduce memory use as the context grows, and that Ollama uses it automatically where the backend and hardware support it.

  3. Use a smaller or more heavily quantised model. Freeing space from the weights leaves more for context. Our GGUF quantisation explainer covers the trade-offs.

  4. Only ask for what you need. A 4k-8k context is enough for most quick questions; save 32k and above for long documents and coding.

To see how much memory a model is really using, follow our guide to checking VRAM usage in Windows. For more speed tips, see how to speed up a local LLM in Ollama, and for buying advice, how much VRAM you need.

Our verdict

Context length is the hidden cost of running AI locally: every extra token of context adds to the KV cache, which competes with the model for VRAM. Ollama defaults to 4k tokens on cards with under 24GiB, so raise it deliberately, check ollama ps to make sure the model stays 100% on the GPU, and use a q8_0 KV cache to roughly halve cache memory if you need more room. Forge PC hasn't benchmarked these settings; the details come from Ollama's and Hugging Face's documentation.

The bottom line

Context length is the hidden cost of running AI locally: every extra token of context adds to the KV cache, which competes with the model for VRAM. Ollama defaults to 4k tokens on cards with under 24GiB, so raise it deliberately, check `ollama ps` to make sure the model stays 100% on the GPU, and use a q8_0 KV cache to roughly halve cache memory if you need more room. Forge PC hasn't benchmarked these settings; the details come from Ollama's and Hugging Face's documentation.

Sources

  1. 01Ollama, Context length
  2. 02Ollama, FAQ
  3. 03Hugging Face, KV cache strategies (Transformers documentation)

About the author

Forge PC Staff

Editorial team

The shared byline for articles researched, written and edited by the Forge PC editorial team. How we source, check and correct our work is set out in our editorial policy.

More from Forge PC Staff

Read next

How to Enable AMD Fluid Motion Frames (AFMF 2.1) on a Radeon Graphics Card

How-To

How to Enable AMD Fluid Motion Frames (AFMF 2.1) on a Radeon Graphics Card

To enable AMD Fluid Motion Frames, install Adrenalin 25.3.1 or newer, then select the HYPR-RX profile (RX 7000 and newer) or turn on AMD Fluid Motion Frames under Gaming > Graphics, globally or per game. AFMF 2.1 works on RX 6000 cards and newer in DirectX 11, DirectX 12, Vulkan and OpenGL games, with VSync off; RX 6000 cards need exclusive full screen.

Forge PC Staff2 min

Two vs Four RAM Sticks for DDR5: Why 2x32GB Usually Beats 4x16GB

Versus

Two vs Four RAM Sticks for DDR5: Why 2x32GB Usually Beats 4x16GB

Two DDR5 sticks are better for most builds. AMD lists DDR5-5600 as the AM5 maximum with two modules but DDR5-3600 with four, and XMP or EXPO speeds above that are overclocking and not guaranteed. Buy 2x16GB or 2x32GB for 32GB or 64GB, and use four sticks, ideally a matched kit, only if you need 128GB or more.

Forge PC Staff3 min

AMD's FSR 4.1.2 Unreal Engine 5 Plugin Brings ML Upscaling to RX 6000 Cards

News

AMD's FSR 4.1.2 Unreal Engine 5 Plugin Brings ML Upscaling to RX 6000 Cards

AMD's FSR Unreal Engine 5 plugin now uses FSR Upscaling 4.1.2, bringing ML-based upscaling to Radeon RX 6000 cards and select Ryzen integrated graphics as well as RX 7000 and RX 9000, in UE 5.3 to 5.8 games that adopt it. It's DirectX 12 only for now and adds Frame Generation 4.0.2 with driver-assisted frame pacing on RX 9000 cards.

Forge PC Staff2 min

NVIDIA's DGX Spark HPC Developer Kit Links Four Sparks Into a 512GB Desk-Side Cluster

News

NVIDIA's DGX Spark HPC Developer Kit Links Four Sparks Into a 512GB Desk-Side Cluster

NVIDIA's DGX Spark HPC Developer Kit, announced at SC26 on 16 November 2026, links four DGX Spark systems in a ring over ConnectX-7 with no network switch, for 512GB of combined memory and 20TB of NVMe storage. NVIDIA says it can spread models of up to 700B parameters across the nodes with NVFP4. No UK kit price yet; Scan sells two-Spark bundles at about £12,845.

Forge PC Staff3 min