Skip to content

How-ToGPU

How to Run Local AI Models With Docker Model Runner on Windows 11

Install Docker Desktop 4.41 or later, go to Settings > AI and turn on Enable Docker Model Runner, plus Enable GPU-backed inference if you have an NVIDIA GPU with driver 576.57 or newer. Then use docker model pull and docker model run. Enable host-side TCP to connect apps via http://localhost:12434/engines/v1, but keep it private: the API has no authentication.

By , Editorial team

3 min read

How to Run Local AI Models With Docker Model Runner on Windows 11

If you already use Docker Desktop for development, you don't need a separate app to run local language models. Docker Model Runner (DMR) is built into Docker Desktop and Docker Engine. It pulls models from Docker Hub or Hugging Face, runs them with llama.cpp by default, and serves them through OpenAI- and Ollama-compatible APIs that other apps can use.

Forge PC hasn't tested Docker Model Runner for this guide. The steps follow Docker's own documentation, checked on 17 November 2026.

What you need

Docker's requirements page lists:

  • Docker Desktop for Windows 4.41 or later (or Docker Engine on Linux).

  • For GPU acceleration on Windows (x64): an NVIDIA GPU with driver 576.57 or newer. On Windows on Arm, Qualcomm Adreno 6xx and later GPUs are supported through OpenCL.

  • On Linux with Docker Engine: CPU, NVIDIA (CUDA), AMD (ROCm) and Vulkan backends are supported. NVIDIA needs driver 575.57.08 or newer.

If your driver is older, update it first using our graphics driver update guide. Radeon owners on Windows may be better served by tools covered in our guide to running local AI on an AMD Radeon GPU.

Step 1: Turn on Docker Model Runner

  1. Open Docker Desktop and go to Settings.

  2. Select the AI tab.

  3. Turn on Enable Docker Model Runner.

  4. If you have a supported NVIDIA GPU, also select Enable GPU-backed inference.

  5. Optional: select Enable host-side TCP support and set a Port if you want apps on your PC (outside containers) to reach the API. Docker's examples use port 12434.

You can now use the docker model command and see your models in the Models tab of the Docker Desktop Dashboard.

Step 2: Pull a model

From a terminal:

docker model pull ai/smollm2:360M-Q4_K_M

That's Docker's own small test model. You can also pull GGUF models straight from Hugging Face, for example:

docker model pull hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF

Or, in Docker Desktop, open Models, choose the Docker Hub tab and select Pull. Models are cached locally after the first download. Pick a size that fits your GPU's memory; our how much VRAM do you need guide explains the maths.

Step 3: Chat with it

docker model run ai/smollm2

This opens an interactive chat in the terminal. In Docker Desktop, go to Models > Local and select the play button instead. Docker says models load into memory only when a request arrives and unload when they're not in use.

Step 4: Manage your models

The docker model command reference includes:

  • docker model list to see models you've pulled

  • docker model ps to see which are running

  • docker model unload to free memory

  • docker model rm to delete a model, or docker model purge to remove them all

  • docker model df to see how much disk space DMR is using

  • docker model logs for troubleshooting

To change a model's context size, Docker's overview page gives the example docker model configure --context-size 8192 ai/qwen2.5-coder. A bigger context uses more memory, as our context length and VRAM explainer shows.

Step 5: Connect other apps

With host-side TCP enabled, Docker lists these base URLs for apps running on your PC:

  • OpenAI-compatible clients: http://localhost:12434/engines/v1

  • Ollama- and Anthropic-compatible clients: http://localhost:12434

From inside a container, use http://model-runner.docker.internal. Docker's integrations docs cover Open WebUI and coding tools such as Continue, Cline and Aider.

Security note: Docker warns that the Model Runner API is not authenticated. Anything that can reach it, including other containers on the same Docker network, can pull, load and run models. Don't expose the port beyond your own machine.

Should you use it instead of Ollama or LM Studio?

If you live in Docker already, DMR keeps models alongside your containers and Compose files, and the Requests tab in Docker Desktop shows each prompt, its context usage and response time. If you just want a desktop chat app, Ollama or LM Studio are simpler to start with.

Our verdict

To run local AI with Docker Model Runner, install Docker Desktop 4.41 or later, turn on Enable Docker Model Runner (and GPU-backed inference with an NVIDIA driver 576.57 or newer) under Settings > AI, then use docker model pull and docker model run. Enable host-side TCP to connect apps through http://localhost:12434/engines/v1, and keep that port private because the API has no authentication. Forge PC hasn't tested it.

The bottom line

To run local AI with Docker Model Runner, install Docker Desktop 4.41 or later, turn on Enable Docker Model Runner (and GPU-backed inference with an NVIDIA driver 576.57 or newer) under Settings > AI, then use `docker model pull` and `docker model run`. Enable host-side TCP to connect apps through `http://localhost:12434/engines/v1`, and keep that port private because the API has no authentication. Forge PC hasn't tested it.

Sources

  1. 01Docker Docs, Docker Model Runner overview and requirements
  2. 02Docker Docs, Get started with Docker Model Runner
  3. 03Docker Docs, Docker Model Runner API reference
  4. 04Docker Docs, docker model CLI reference

About the author

Forge PC Staff

Editorial team

The shared byline for articles researched, written and edited by the Forge PC editorial team. How we source, check and correct our work is set out in our editorial policy.

More from Forge PC Staff

Read next

How to Optimise an SSD in Windows 11: Check TRIM, Run Optimize Drives and Skip the Myths

How-To

How to Optimise an SSD in Windows 11: Check TRIM, Run Optimize Drives and Skip the Myths

Windows 11 trims SSDs automatically, weekly by default. Run fsutil behavior query DisableDeleteNotify as administrator: 0 means TRIM is on. Check that scheduled optimisation is on in Defragment and Optimize Drives, and use Optimize or Optimize-Volume -ReTrim to run it by hand. Never defragment an SSD with third-party tools.

Forge PC Staff3 min

How to Turn On Hardware-Accelerated GPU Scheduling in Windows 11 (and When to Turn It Off)

How-To

How to Turn On Hardware-Accelerated GPU Scheduling in Windows 11 (and When to Turn It Off)

Open Settings > System > Display > Graphics > Advanced graphics settings, turn on Hardware-accelerated GPU scheduling and restart. NVIDIA says DLSS Frame Generation needs it, and CD PROJEKT RED says it's on by default in Windows 11. If the toggle is missing, update your GPU driver; Microsoft says most users won't notice a significant change otherwise.

Forge PC Staff3 min

Project Maya Runs 321B to 748B MoE Models on One GPU by Spreading Them Across VRAM, RAM and SSD

News

Project Maya Runs 321B to 748B MoE Models on One GPU by Spreading Them Across VRAM, RAM and SSD

Project Maya is a free, open-source engine that runs GLM-5.3 (744B), GLM-5.3-Flash (321B), DeepSeek-V4.1-Flash (748B) and MiniMax-M3.1-Flash (428B) on desktop GPUs by keeping the busiest experts in VRAM and the rest in RAM and on an NVMe SSD. It needs an RTX 20-series or newer card (or a supported Radeon), 32GB+ of RAM and 75-350GB of SSD space.

Forge PC Staff4 min

How to Enable AMD Fluid Motion Frames (AFMF 2.1) on a Radeon Graphics Card

How-To

How to Enable AMD Fluid Motion Frames (AFMF 2.1) on a Radeon Graphics Card

To enable AMD Fluid Motion Frames, install Adrenalin 25.3.1 or newer, then select the HYPR-RX profile (RX 7000 and newer) or turn on AMD Fluid Motion Frames under Gaming > Graphics, globally or per game. AFMF 2.1 works on RX 6000 cards and newer in DirectX 11, DirectX 12, Vulkan and OpenGL games, with VSync off; RX 6000 cards need exclusive full screen.

Forge PC Staff2 min