Skip to content

How-ToGPU

How to Install Ollama on Linux and Run Local AI Models on Your GPU

How to install Ollama on Linux with the official script or manually, set it up as a systemd service, install CUDA or ROCm drivers for NVIDIA and AMD GPUs, run and manage models, change settings, update, read logs and uninstall.

By , Systems & Technical Editor

4 min read

How to Install Ollama on Linux and Run Local AI Models on Your GPU

Ollama is one of the easiest ways to run large language models locally, and on Linux it installs with a single command. It's a good fit for a dual-boot gaming PC, a home server or an old PC with a decent graphics card. This guide follows Ollama's official Linux documentation.

What you need

  • A 64-bit Linux install. Ollama publishes packages for x86-64 (amd64) and ARM64.

  • Optional: an NVIDIA or AMD graphics card. Ollama's Linux guide covers optional CUDA (NVIDIA) and ROCm (AMD) driver setup for GPU acceleration. Our guide to how much VRAM you need for local AI explains sizing, and our best GPU for local LLMs guide covers what to buy.

  • Terminal access with sudo rights.

If you're still choosing a Linux distribution, our guide to dual-booting Windows 11 and Linux covers setting one up alongside Windows.

Option 1: Install with the official script (easiest)

Ollama's documentation gives one command to install it:

curl -fsSL https://ollama.com/install.sh | sh

As with any script piped into a shell, you can download it first and read it before running it if you prefer.

Option 2: Manual install

If you'd rather not run the script, Ollama documents a manual install. If you're upgrading from an older version, first remove the old libraries with sudo rm -rf /usr/lib/ollama. Then download and extract the package:

curl -fsSL https://ollama.com/download/ollama-linux-amd64.tar.zst | sudo tar x -C /usr

For an AMD graphics card, Ollama says to also extract the ROCm package:

curl -fsSL https://ollama.com/download/ollama-linux-amd64-rocm.tar.zst | sudo tar x -C /usr

ARM64 systems use the ollama-linux-arm64.tar.zst package instead.

Start Ollama with ollama serve, then in another terminal check it's working with ollama -v.

Ollama recommends adding it as a startup service. Create a user and group for it:

sudo useradd -r -s /bin/false -U -m -d /usr/share/ollama ollama
sudo usermod -a -G ollama $(whoami)

Then create /etc/systemd/system/ollama.service with the service definition from Ollama's documentation (it runs /usr/bin/ollama serve as the ollama user and restarts automatically), and enable it:

sudo systemctl daemon-reload
sudo systemctl enable ollama

Step 2: Set up GPU drivers

  • NVIDIA: Ollama's documentation says to download and install CUDA, then check the driver works by running nvidia-smi, which should print details about your GPU.

  • AMD: Ollama says to install ROCm v7. It notes that the amdgpu driver in the mainline Linux kernel is older and may not support all ROCm features, and recommends installing the latest driver from AMD's Linux drivers page for the best support of Radeon GPUs.

Our CUDA vs ROCm vs Vulkan explainer covers the differences between the GPU back ends.

Step 3: Start Ollama and run a model

Start the service and check its status:

sudo systemctl start ollama
sudo systemctl status ollama

Then run a model. Ollama's CLI reference uses this example:

ollama run gemma4

Models take up disk space, so check you have room first (our guide to how much storage local AI needs helps), and you can download a model in advance with ollama pull. Other useful commands from the CLI reference are ollama pull to download a model, ollama ls to list installed models, ollama ps to list running models, ollama stop to stop one and ollama rm to remove one. Our guide to managing Ollama models covers these in more detail.

Step 4: Customise, update and check logs

  • Settings: Ollama is configured with environment variables. Run sudo systemctl edit ollama to edit the service, or create /etc/systemd/system/ollama.service.d/override.conf with lines such as Environment="OLLAMA_DEBUG=1" under a [Service] heading.

  • Updating: run the install script again, or re-download the package as in the manual install.

  • Specific versions: set OLLAMA_VERSION when running the install script, for example curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=0.5.7 sh, which is the example version in Ollama's documentation.

  • Logs: journalctl -e -u ollama shows the logs for Ollama running as a startup service, which is a good place to start if something isn't working.

Uninstalling

Ollama's documentation lists the steps: stop and disable the service, remove the service file, remove the Ollama libraries and binary, and delete the ollama user, group and /usr/share/ollama folder (which holds downloaded models). Follow the exact commands on Ollama's Linux page.

Our verdict

On Linux, the official install script is the quickest way to get Ollama running, and the manual route gives you more control with a systemd service. Install CUDA or ROCm first so Ollama can use your graphics card, then use ollama run to download and chat with a model. If you want to use it from other PCs on your network, see our guide to sharing Ollama on your local network.

The bottom line

On Linux, the official install script is the quickest way to get Ollama running, and the manual route gives you more control with a systemd service. Install CUDA or ROCm first so Ollama can use your graphics card, then use `ollama run` to download and chat with a model. If you want to use it from other PCs on your network, see our guide to sharing Ollama on your local network.

Sources

  1. 01Ollama, Linux (official documentation)
  2. 02Ollama, CLI Reference

About the author

Marcus Thorne

Systems & Technical Editor

Marcus writes Forge PC's step-by-step build, troubleshooting and system optimisation guides. His background spans Windows kernel and power-state diagnostics, UEFI/BIOS configuration, NVMe storage arrays, ATX 3.1 power delivery and running local AI and LLM workloads on consumer graphics cards.

More from Marcus Thorne

Read next

ComfyUI vs SwarmUI: Which Local AI Image Generator Should You Use?

Versus

ComfyUI vs SwarmUI: Which Local AI Image Generator Should You Use?

ComfyUI's node graph gives full control over local AI image, video and audio workflows; SwarmUI adds a simpler Generate tab and can run ComfyUI as its back end, with the graph a tab away. SwarmUI suits beginners, ComfyUI power users.

Callum Reid4 min