When people compare graphics cards for local AI, the number that matters most is tokens per second (t/s): how quickly a model reads your prompt and how quickly it writes its answer. A model that generates 5 t/s feels sluggish; one that generates many times that feels instant.
You don't need to rely on other people's charts. This guide shows how to measure speed on your own PC with two free tools: llama.cpp's llama-bench and the timing data that Ollama returns with every response. Forge PC hasn't published its own LLM benchmarks; this guide explains the method, using llama.cpp's and Ollama's documentation.
Two numbers: prompt processing and generation
Local LLM speed has two parts, and llama-bench measures them separately:
Prompt processing (pp): how fast the model reads your input. This matters for long prompts, documents and chat history.
Text generation (tg): how fast the model writes new tokens. This is the speed you feel while watching an answer appear.
llama-bench also has a combined pg test that processes a prompt and then generates text.
Method 1: llama-bench (most accurate)
llama-bench is one of the tools in llama.cpp, the open-source inference engine; we compare it with Ollama in llama.cpp vs Ollama. Get a llama.cpp build that supports your GPU and you'll find llama-bench alongside the other tools. You also need a model in GGUF format.
Run a basic test
From the llama.cpp folder, run:
llama-bench -m C:\models\your-model.gguf
By default, according to the llama-bench documentation, it runs a 512-token prompt processing test and a 128-token generation test, repeats each test five times and reports the average tokens per second with a standard deviation. The output is a table showing the model, its size, parameter count, backend, number of GPU layers (ngl) and the result for each test.
The documentation notes that llama-bench's measurements don't include tokenisation and sampling time, so real chats will feel slightly slower than the raw figure.
Useful options
-p 512 -n 128 sets the prompt and generation lengths. Multiple values can be separated by commas, for example -n 128,256,512.
-ngl sets how many layers go on the GPU (the output table shows it as ngl). Lowering it shows how much speed you lose when part of a model sits in system RAM.
-d 4096 runs the test at a context depth of 4,096 tokens by pre-filling the cache. This shows how speed changes once a conversation is already long.
-fa on or -fa off toggles flash attention (the default is auto).
-ctk and -ctv set the KV cache type, so you can test whether a quantised cache changes speed.
-r 10 raises the number of repetitions for steadier results.
-o csv, -o json or -o md changes the output format so you can paste results into a spreadsheet.
You can give several values at once, and llama-bench will test every combination. That makes it easy to compare, say, full GPU offload against partial offload in one run.
Method 2: Ollama's built-in timings
If you use Ollama, you can calculate speed from data it already returns. Ollama's API documentation says responses include:
prompt_eval_count: the number of input tokens.
prompt_eval_duration: how long it took to evaluate the uncached prompt tokens.
eval_count: the number of output tokens.
eval_duration: how long it took to generate them.
load_duration and total_duration: model load time and total time.
All timings are in nanoseconds. So:
Generation speed = eval_count / eval_duration x 1,000,000,000
Prompt speed = prompt_eval_count / prompt_eval_duration x 1,000,000,000
For example, in Windows PowerShell:
$r = Invoke-RestMethod -Uri http://localhost:11434/api/generate -Method Post -Body '{"model":"gpt-oss:20b","prompt":"Explain PCIe lanes in 200 words.","stream":false}'
$r.eval_count / $r.eval_duration * 1e9
Run it a few times and ignore the first result, which includes loading the model into memory (shown in load_duration). Then run ollama ps to confirm where the model is loaded; Ollama's FAQ says "100% GPU" means it's fully in VRAM.
How to make comparisons fair
Speeds are only comparable if everything except the thing you're testing stays the same:
Same model file and quantisation. A Q4 and a Q8 version of the same model will give very different results. See LLM quantisation explained.
Same prompt and generation lengths, and same context depth.
Same backend and settings, such as GPU layers and flash attention.
Close other GPU apps, including games and browsers with hardware acceleration.
Repeat runs and look at the standard deviation llama-bench reports; large swings suggest something else is competing for the GPU.
Watch your VRAM while testing, too; our guide to checking VRAM usage in Windows shows how. A model that spills out of VRAM into system RAM will be slower.
What to do with your results
Our verdict
To benchmark a local LLM, run llama.cpp's llama-bench against a GGUF model: by default it tests 512-token prompt processing and 128-token generation, repeated five times, and reports average tokens per second. In Ollama, divide eval_count by eval_duration (in nanoseconds) for generation speed. Keep the model, quantisation, lengths and settings identical when comparing hardware, and repeat each run. Forge PC hasn't published its own LLM benchmarks.