Skip to content

How-ToRAM

How to Use Embedding Models in Ollama for Local Semantic Search and RAG

Embedding models turn text into vectors for semantic search and RAG. Pull one such as embeddinggemma in Ollama, generate vectors with ollama run or the /api/embed endpoint, store them, then embed questions with the same model and compare using cosine similarity, all locally.

By , Editorial team

3 min read

How to Use Embedding Models in Ollama for Local Semantic Search and RAG

Chat models get the attention, but if you want a local AI that can answer questions about your documents, you also need an embedding model. Ollama runs these locally alongside its chat models. This guide covers what embeddings are, how to generate them with Ollama's command line and API, and how they fit into a local search or RAG (retrieval-augmented generation) setup. It's based on Ollama's official documentation; Forge PC hasn't benchmarked the models mentioned.

What an embedding model does

Ollama's documentation describes embeddings as a way to turn text into numeric vectors that you can store in a vector database, search with cosine similarity, or use in RAG pipelines. Text with a similar meaning ends up with similar vectors, so you can find a relevant paragraph even when it doesn't share the exact words of your question. Ollama says the vector length depends on the model, typically 384 to 1,024 dimensions.

Embedding models are small compared with chat models. The examples in Ollama's embedding blog post include mxbai-embed-large (334M parameters) and nomic-embed-text (137M parameters), and the current docs use embeddinggemma in their examples.

If you haven't installed Ollama yet, start with our guide to running AI models locally with Ollama and LM Studio.

Step 1: Pull an embedding model

Open a terminal and download a model, for example:

ollama pull embeddinggemma

Our guide to managing Ollama models explains how to list and remove models later.

Step 2: Generate an embedding from the command line

Ollama's docs show that you can run an embedding model directly:

ollama run embeddinggemma "Hello world"

You can also pipe text in, for example echo "Hello world" | ollama run embeddinggemma. The output is a JSON array of numbers: the vector.

Step 3: Use the API

For anything practical you'll call Ollama's local API from a script or app. Ollama's docs give this cURL example for the /api/embed endpoint:

curl -X POST http://localhost:11434/api/embed \
  -H "Content-Type: application/json" \
  -d '{
    "model": "embeddinggemma",
    "input": "The quick brown fox jumps over the lazy dog."
  }'
  • Batches: pass an array of strings as input to embed several pieces of text in one request.

  • Normalised output: Ollama notes that /api/embed returns L2-normalised (unit-length) vectors.

  • Libraries: the official Python and JavaScript libraries provide an embed function that does the same thing.

If your app expects an OpenAI-style endpoint instead, see how to use Ollama's OpenAI-compatible API.

The basic recipe is the same whatever tool you use:

  1. Split your documents into chunks (a few paragraphs each).

  2. Embed every chunk and store the vectors, for example in a vector database, as Ollama's docs suggest.

  3. When you ask a question, embed the question with the same model.

  4. Find the chunks whose vectors are closest, using cosine similarity, and show them, or pass them to a chat model as context. That last step is RAG.

Ollama's two tips are worth repeating: use cosine similarity for most semantic search, and use the same embedding model for indexing and querying. If you'd rather not write code, a document-chat app may be easier: see how to chat with your documents in AnythingLLM and our Open WebUI vs AnythingLLM comparison.

Hardware: what you need

The example embedding models in Ollama's blog have 137M and 334M parameters, far fewer than typical local chat models, so the chat model you pair them with is likely to matter more for your hardware. For running everything without a graphics card, see how to run a local LLM without a GPU.

Our verdict

Pull an embedding model such as embeddinggemma, send text to Ollama's /api/embed endpoint and store the vectors. Then embed each question with the same model and compare with cosine similarity. That's the core of private, local semantic search and RAG, and it runs alongside the chat models you already use in Ollama.

The bottom line

Pull an embedding model such as embeddinggemma, send text to Ollama's /api/embed endpoint and store the vectors. Then embed each question with the same model and compare with cosine similarity. That's the core of private, local semantic search and RAG, and it runs alongside the chat models you already use in Ollama.

Sources

  1. 01Ollama Docs, Embeddings
  2. 02Ollama Blog, Embedding models

About the author

Forge PC Staff

Editorial team

The shared byline for articles researched, written and edited by the Forge PC editorial team. How we source, check and correct our work is set out in our editorial policy.

More from Forge PC Staff

Read next

How to Turn Thinking On or Off in Ollama (DeepSeek-R1, Qwen 3, gpt-oss)

How-To

How to Turn Thinking On or Off in Ollama (DeepSeek-R1, Qwen 3, gpt-oss)

Ollama lets you control reasoning models' thinking: --think and --think=false on the command line, /set think and /set nothink in chats, --hidethinking for scripts and a think field in the API. Check /api/show first, as models such as gpt-oss use low, medium and high levels instead.

Forge PC Staff2 min

G.SKILL Trident Z5 Neo RGB vs Trident Z5 RGB: Which DDR5 Kit Should You Buy?

Versus

G.SKILL Trident Z5 Neo RGB vs Trident Z5 RGB: Which DDR5 Kit Should You Buy?

G.SKILL's Trident Z5 Neo RGB is its AMD EXPO kit and the Trident Z5 RGB its Intel XMP kit. The white 32GB Neo is rated DDR5-6000 CL30 at 1.35V with EXPO and XMP; the white Trident Z5 RGB is rated DDR5-6400 CL32 at 1.40V with XMP only. Both work out at about 10ns first-word latency, so choose by platform.

Forge PC Staff3 min

Best White DDR5 RAM Kits 2026: Picks for White and Snow-Themed PC Builds

Best Picks

Best White DDR5 RAM Kits 2026: Picks for White and Snow-Themed PC Builds

Our top white DDR5 kit is the G.SKILL Trident Z5 Neo RGB 32GB White, rated DDR5-6000 CL30 at 1.35V with EXPO and XMP. Corsair's Vengeance White suits non-RGB builds, the Vengeance RGB and Trident Z5 RGB White suit Intel, and the 64GB Trident Z5 Neo RGB or 96GB Flare X5 White add capacity.

Forge PC Staff4 min