Skip to content

How-ToGPU

How to Benchmark Local LLM Speed: Measure Tokens per Second with llama-bench and Ollama

To benchmark a local LLM, run llama.cpp's llama-bench on a GGUF model: by default it tests 512-token prompt processing and 128-token generation five times and reports tokens per second. In Ollama, divide eval_count by eval_duration. Keep model, quantisation and settings identical when comparing hardware.

By , Editorial team

4 min read

How to Benchmark Local LLM Speed: Measure Tokens per Second with llama-bench and Ollama

When people compare graphics cards for local AI, the number that matters most is tokens per second (t/s): how quickly a model reads your prompt and how quickly it writes its answer. A model that generates 5 t/s feels sluggish; one that generates many times that feels instant.

You don't need to rely on other people's charts. This guide shows how to measure speed on your own PC with two free tools: llama.cpp's llama-bench and the timing data that Ollama returns with every response. Forge PC hasn't published its own LLM benchmarks; this guide explains the method, using llama.cpp's and Ollama's documentation.

Two numbers: prompt processing and generation

Local LLM speed has two parts, and llama-bench measures them separately:

  • Prompt processing (pp): how fast the model reads your input. This matters for long prompts, documents and chat history.

  • Text generation (tg): how fast the model writes new tokens. This is the speed you feel while watching an answer appear.

llama-bench also has a combined pg test that processes a prompt and then generates text.

Method 1: llama-bench (most accurate)

llama-bench is one of the tools in llama.cpp, the open-source inference engine; we compare it with Ollama in llama.cpp vs Ollama. Get a llama.cpp build that supports your GPU and you'll find llama-bench alongside the other tools. You also need a model in GGUF format.

Run a basic test

From the llama.cpp folder, run:

llama-bench -m C:\models\your-model.gguf

By default, according to the llama-bench documentation, it runs a 512-token prompt processing test and a 128-token generation test, repeats each test five times and reports the average tokens per second with a standard deviation. The output is a table showing the model, its size, parameter count, backend, number of GPU layers (ngl) and the result for each test.

The documentation notes that llama-bench's measurements don't include tokenisation and sampling time, so real chats will feel slightly slower than the raw figure.

Useful options

  • -p 512 -n 128 sets the prompt and generation lengths. Multiple values can be separated by commas, for example -n 128,256,512.

  • -ngl sets how many layers go on the GPU (the output table shows it as ngl). Lowering it shows how much speed you lose when part of a model sits in system RAM.

  • -d 4096 runs the test at a context depth of 4,096 tokens by pre-filling the cache. This shows how speed changes once a conversation is already long.

  • -fa on or -fa off toggles flash attention (the default is auto).

  • -ctk and -ctv set the KV cache type, so you can test whether a quantised cache changes speed.

  • -r 10 raises the number of repetitions for steadier results.

  • -o csv, -o json or -o md changes the output format so you can paste results into a spreadsheet.

You can give several values at once, and llama-bench will test every combination. That makes it easy to compare, say, full GPU offload against partial offload in one run.

Method 2: Ollama's built-in timings

If you use Ollama, you can calculate speed from data it already returns. Ollama's API documentation says responses include:

  • prompt_eval_count: the number of input tokens.

  • prompt_eval_duration: how long it took to evaluate the uncached prompt tokens.

  • eval_count: the number of output tokens.

  • eval_duration: how long it took to generate them.

  • load_duration and total_duration: model load time and total time.

All timings are in nanoseconds. So:

  • Generation speed = eval_count / eval_duration x 1,000,000,000

  • Prompt speed = prompt_eval_count / prompt_eval_duration x 1,000,000,000

For example, in Windows PowerShell:

$r = Invoke-RestMethod -Uri http://localhost:11434/api/generate -Method Post -Body '{"model":"gpt-oss:20b","prompt":"Explain PCIe lanes in 200 words.","stream":false}'
$r.eval_count / $r.eval_duration * 1e9

Run it a few times and ignore the first result, which includes loading the model into memory (shown in load_duration). Then run ollama ps to confirm where the model is loaded; Ollama's FAQ says "100% GPU" means it's fully in VRAM.

How to make comparisons fair

Speeds are only comparable if everything except the thing you're testing stays the same:

  1. Same model file and quantisation. A Q4 and a Q8 version of the same model will give very different results. See LLM quantisation explained.

  2. Same prompt and generation lengths, and same context depth.

  3. Same backend and settings, such as GPU layers and flash attention.

  4. Close other GPU apps, including games and browsers with hardware acceleration.

  5. Repeat runs and look at the standard deviation llama-bench reports; large swings suggest something else is competing for the GPU.

Watch your VRAM while testing, too; our guide to checking VRAM usage in Windows shows how. A model that spills out of VRAM into system RAM will be slower.

What to do with your results

Our verdict

To benchmark a local LLM, run llama.cpp's llama-bench against a GGUF model: by default it tests 512-token prompt processing and 128-token generation, repeated five times, and reports average tokens per second. In Ollama, divide eval_count by eval_duration (in nanoseconds) for generation speed. Keep the model, quantisation, lengths and settings identical when comparing hardware, and repeat each run. Forge PC hasn't published its own LLM benchmarks.

The bottom line

To benchmark a local LLM, run llama.cpp's llama-bench against a GGUF model: by default it tests 512-token prompt processing and 128-token generation, repeated five times, and reports average tokens per second. In Ollama, divide `eval_count` by `eval_duration` (in nanoseconds) for generation speed. Keep the model, quantisation, lengths and settings identical when comparing hardware, and repeat each run. Forge PC hasn't published its own LLM benchmarks.

Sources

  1. 01ggml-org/llama.cpp on GitHub, llama-bench README
  2. 02Ollama Docs, Usage (API response metrics)
  3. 03Ollama Docs, Local FAQ

About the author

Forge PC Staff

Editorial team

The shared byline for articles researched, written and edited by the Forge PC editorial team. How we source, check and correct our work is set out in our editorial policy.

More from Forge PC Staff

Read next

How to Enable AMD Fluid Motion Frames (AFMF 2.1) on a Radeon Graphics Card

How-To

How to Enable AMD Fluid Motion Frames (AFMF 2.1) on a Radeon Graphics Card

To enable AMD Fluid Motion Frames, install Adrenalin 25.3.1 or newer, then select the HYPR-RX profile (RX 7000 and newer) or turn on AMD Fluid Motion Frames under Gaming > Graphics, globally or per game. AFMF 2.1 works on RX 6000 cards and newer in DirectX 11, DirectX 12, Vulkan and OpenGL games, with VSync off; RX 6000 cards need exclusive full screen.

Forge PC Staff2 min

How to Use System Restore in Windows 11: Turn It On, Create a Restore Point and Roll Back

How-To

How to Use System Restore in Windows 11: Turn It On, Create a Restore Point and Roll Back

Microsoft says System Protection isn't on by default. Search for Create a restore point, select your C: drive, choose Configure and turn it on, then create a restore point before installing drivers or tools. To roll back, run rstrui.exe, or use Troubleshoot > Advanced options > System Restore in Windows RE. Personal files aren't affected, but it isn't a backup.

Forge PC Staff3 min

How to Use Live Captions and Translation in Windows 11

How-To

How to Use Live Captions and Translation in Windows 11

Press Windows + Ctrl + L to turn on live captions in Windows 11 22H2 or later, download the language files, then dock or float the captions and pick a style. Captions are generated on the device and work offline. On a Copilot+ PC with 24H2 or later, live captions can translate from over 40 languages into English and 27 into Chinese (Simplified).

Forge PC Staff3 min

How to Manage Ollama Models: List, Remove and Move Them to Another Drive

How-To

How to Manage Ollama Models: List, Remove and Move Them to Another Drive

Run ollama ls to list your models, ollama ps to see what's loaded, ollama stop to unload one and ollama rm to delete it. On Windows, models are stored in C:\Users\%username%\.ollama\models; to move them, quit Ollama, set an OLLAMA_MODELS user environment variable to a folder on a bigger drive and restart Ollama.

Forge PC Staff3 min