Chat models get the attention, but if you want a local AI that can answer questions about your documents, you also need an embedding model. Ollama runs these locally alongside its chat models. This guide covers what embeddings are, how to generate them with Ollama's command line and API, and how they fit into a local search or RAG (retrieval-augmented generation) setup. It's based on Ollama's official documentation; Forge PC hasn't benchmarked the models mentioned.
What an embedding model does
Ollama's documentation describes embeddings as a way to turn text into numeric vectors that you can store in a vector database, search with cosine similarity, or use in RAG pipelines. Text with a similar meaning ends up with similar vectors, so you can find a relevant paragraph even when it doesn't share the exact words of your question. Ollama says the vector length depends on the model, typically 384 to 1,024 dimensions.
Embedding models are small compared with chat models. The examples in Ollama's embedding blog post include mxbai-embed-large (334M parameters) and nomic-embed-text (137M parameters), and the current docs use embeddinggemma in their examples.
If you haven't installed Ollama yet, start with our guide to running AI models locally with Ollama and LM Studio.
Step 1: Pull an embedding model
Open a terminal and download a model, for example:
ollama pull embeddinggemma
Our guide to managing Ollama models explains how to list and remove models later.
Step 2: Generate an embedding from the command line
Ollama's docs show that you can run an embedding model directly:
ollama run embeddinggemma "Hello world"
You can also pipe text in, for example echo "Hello world" | ollama run embeddinggemma. The output is a JSON array of numbers: the vector.
Step 3: Use the API
For anything practical you'll call Ollama's local API from a script or app. Ollama's docs give this cURL example for the /api/embed endpoint:
curl -X POST http://localhost:11434/api/embed \
-H "Content-Type: application/json" \
-d '{
"model": "embeddinggemma",
"input": "The quick brown fox jumps over the lazy dog."
}'
Batches: pass an array of strings as input to embed several pieces of text in one request.
Normalised output: Ollama notes that /api/embed returns L2-normalised (unit-length) vectors.
Libraries: the official Python and JavaScript libraries provide an embed function that does the same thing.
If your app expects an OpenAI-style endpoint instead, see how to use Ollama's OpenAI-compatible API.
Step 4: Build simple semantic search
The basic recipe is the same whatever tool you use:
Split your documents into chunks (a few paragraphs each).
Embed every chunk and store the vectors, for example in a vector database, as Ollama's docs suggest.
When you ask a question, embed the question with the same model.
Find the chunks whose vectors are closest, using cosine similarity, and show them, or pass them to a chat model as context. That last step is RAG.
Ollama's two tips are worth repeating: use cosine similarity for most semantic search, and use the same embedding model for indexing and querying. If you'd rather not write code, a document-chat app may be easier: see how to chat with your documents in AnythingLLM and our Open WebUI vs AnythingLLM comparison.
Hardware: what you need
The example embedding models in Ollama's blog have 137M and 334M parameters, far fewer than typical local chat models, so the chat model you pair them with is likely to matter more for your hardware. For running everything without a graphics card, see how to run a local LLM without a GPU.
Our verdict
Pull an embedding model such as embeddinggemma, send text to Ollama's /api/embed endpoint and store the vectors. Then embed each question with the same model and compare with cosine similarity. That's the core of private, local semantic search and RAG, and it runs alongside the chat models you already use in Ollama.