If you already use Docker Desktop for development, you don't need a separate app to run local language models. Docker Model Runner (DMR) is built into Docker Desktop and Docker Engine. It pulls models from Docker Hub or Hugging Face, runs them with llama.cpp by default, and serves them through OpenAI- and Ollama-compatible APIs that other apps can use.
Forge PC hasn't tested Docker Model Runner for this guide. The steps follow Docker's own documentation, checked on 17 November 2026.
What you need
Docker's requirements page lists:
Docker Desktop for Windows 4.41 or later (or Docker Engine on Linux).
For GPU acceleration on Windows (x64): an NVIDIA GPU with driver 576.57 or newer. On Windows on Arm, Qualcomm Adreno 6xx and later GPUs are supported through OpenCL.
On Linux with Docker Engine: CPU, NVIDIA (CUDA), AMD (ROCm) and Vulkan backends are supported. NVIDIA needs driver 575.57.08 or newer.
If your driver is older, update it first using our graphics driver update guide. Radeon owners on Windows may be better served by tools covered in our guide to running local AI on an AMD Radeon GPU.
Step 1: Turn on Docker Model Runner
Open Docker Desktop and go to Settings.
Select the AI tab.
Turn on Enable Docker Model Runner.
If you have a supported NVIDIA GPU, also select Enable GPU-backed inference.
Optional: select Enable host-side TCP support and set a Port if you want apps on your PC (outside containers) to reach the API. Docker's examples use port 12434.
You can now use the docker model command and see your models in the Models tab of the Docker Desktop Dashboard.
Step 2: Pull a model
From a terminal:
docker model pull ai/smollm2:360M-Q4_K_M
That's Docker's own small test model. You can also pull GGUF models straight from Hugging Face, for example:
docker model pull hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF
Or, in Docker Desktop, open Models, choose the Docker Hub tab and select Pull. Models are cached locally after the first download. Pick a size that fits your GPU's memory; our how much VRAM do you need guide explains the maths.
Step 3: Chat with it
docker model run ai/smollm2
This opens an interactive chat in the terminal. In Docker Desktop, go to Models > Local and select the play button instead. Docker says models load into memory only when a request arrives and unload when they're not in use.
Step 4: Manage your models
The docker model command reference includes:
docker model list to see models you've pulled
docker model ps to see which are running
docker model unload to free memory
docker model rm to delete a model, or docker model purge to remove them all
docker model df to see how much disk space DMR is using
docker model logs for troubleshooting
To change a model's context size, Docker's overview page gives the example docker model configure --context-size 8192 ai/qwen2.5-coder. A bigger context uses more memory, as our context length and VRAM explainer shows.
Step 5: Connect other apps
With host-side TCP enabled, Docker lists these base URLs for apps running on your PC:
From inside a container, use http://model-runner.docker.internal. Docker's integrations docs cover Open WebUI and coding tools such as Continue, Cline and Aider.
Security note: Docker warns that the Model Runner API is not authenticated. Anything that can reach it, including other containers on the same Docker network, can pull, load and run models. Don't expose the port beyond your own machine.
Should you use it instead of Ollama or LM Studio?
If you live in Docker already, DMR keeps models alongside your containers and Compose files, and the Requests tab in Docker Desktop shows each prompt, its context usage and response time. If you just want a desktop chat app, Ollama or LM Studio are simpler to start with.
Our verdict
To run local AI with Docker Model Runner, install Docker Desktop 4.41 or later, turn on Enable Docker Model Runner (and GPU-backed inference with an NVIDIA driver 576.57 or newer) under Settings > AI, then use docker model pull and docker model run. Enable host-side TCP to connect apps through http://localhost:12434/engines/v1, and keep that port private because the API has no authentication. Forge PC hasn't tested it.