Embeddings in Production: Vector Search Without the Server

Sooner or later, every serious AI product needs retrieval: answering questions from a client's own documents, deduplicating content, grounding a model in facts it wasn't trained on. The building blocks are always the same: embeddings on one side, a vector store on the other. Here is how we approach both in production, why we generate embeddings on-premise, and why we picked zvec over running yet another database server.

What embeddings actually are

An embedding model turns a piece of text into a list of numbers, a vector. Not a random list: the model is trained so that texts with similar meaning end up close to each other in that numeric space, even when they share no words. "How do I reset my password?" and "I can't log into my account" land near each other; an invoice about steel pipes lands far away from both.

That single property changes what search can do. Classic keyword search finds documents that contain your words; embedding search finds documents that mean what you meant. In practice you embed all your documents once, store the vectors, then at query time embed the question and look for the nearest vectors, typically by cosine similarity, a measure of the angle between two vectors. The vectors themselves are just arrays of a few hundred to a few thousand floats (their dimension); the intelligence lives in the model that produced them.

Two practical rules follow. First, similarity scores are only comparable between vectors produced by the same model: mixing models (or dimensions) in one index makes the numbers meaningless. Second, retrieval quality is decided by the embedding model far more than by the store that indexes the vectors. Choose the model first.

Generating embeddings on-premise

Most teams default to a cloud embeddings API. For much of our work that is simply not an option: the documents being embedded are the client's internal knowledge, and it stays on their infrastructure. The good news is that self-hosting an embedding model in 2026 is almost boring.

On one of our platforms, embeddings are served by a single standalone Docker container running Ollama, CPU only, no GPU, with qwen3-embedding:0.6b, a small multilingual model producing 1024-dimensional vectors. With OLLAMA_KEEP_ALIVE=-1 the model stays resident in memory and the whole thing sits at roughly 1 to 1.5 GB of RAM on our reference deployment. The application talks to it over a plain HTTP endpoint (/api/embed), batching documents in one call. One detail worth knowing about this model family: the retrieval instruct prefix goes on the query side only; documents are encoded raw.

That container replaces an external API entirely: no per-token bill, no rate limits, and not a single byte of client content leaving the server. And if retrieval quality ever needs a bump, the same family ships 4B and 8B variants that run on the same box. Re-embed, swap the model tag, done.

Here is the entire setup, because there is genuinely nothing more to it:

# docker-compose.yml : the entire embedding infrastructure
services:
  embeddings:
    image: ollama/ollama
    volumes:
      - embeddings-models:/root/.ollama
    environment:
      - OLLAMA_KEEP_ALIVE=-1   # keep the model resident in RAM
    ports:
      - "11434:11434"
    restart: unless-stopped
volumes:
  embeddings-models:

docker compose up -d
docker compose exec embeddings ollama pull qwen3-embedding:0.6b

# embed a batch of documents
curl -s http://localhost:11434/api/embed \
  -d '{"model": "qwen3-embedding:0.6b", "input": ["first chunk", "second chunk"]}'

That is the complete embedding infrastructure: a stock image, a volume for the weights, one environment variable. No Dockerfile to maintain, no GPU driver, nothing to build.

Picking a vector store: the landscape

Once you have vectors, you need somewhere to search them. The ecosystem splits into two families: servers you operate (another process, another port, another thing to monitor) and libraries that run inside your application, the way SQLite does for relational data.

Why we put zvec on top

zvec is Alibaba's open-source (Apache 2.0) in-process vector database, built on Proxima, the vector engine behind Alibaba's own production similarity search. For the deployments we do (on-premise, hardened, often air-gapped environments where every extra service is a negotiation), the in-process model is not a nice-to-have. It is the whole point:

• Nothing extra to operate. No port, no cluster, no credentials, no additional attack surface to justify in a security review. The index is files on a volume inside your app's container.
• Hybrid search in one query. Dense and sparse vector similarity, native full-text search on string fields (added in v0.5) and scalar filters, fused in a single MultiQuery. Most embedded options give you only one of these, and real products need several at once.
• Engineered for durability. Write-ahead logging means a process crash doesn't lose writes; reads stay concurrent across processes.
• Headroom. The project's published benchmarks (about 10M vectors indexed in an hour, more than 8,500 QPS on the Cohere 10M dataset) are far beyond what a typical enterprise knowledge base needs.

A word on maturity, because your senior engineers will ask. The Proxima engine has years of production behind it at Alibaba; the zvec packaging around it is young (open-sourced in December 2025, more than 12,000 GitHub stars within months of release). We treat it accordingly: zvec accelerates retrieval in our deployments, but it never owns the data. The next section shows what that means concretely.

And to be clear about the frame: if you need a shared index served to dozens of microservices, a server like Qdrant remains the right architecture. Our claim is narrower: for retrieval that lives inside one application, on infrastructure you don't fully control, the library model wins on every axis we care about.

zvec vs Qdrant

Qdrant is the stronger choice when several services must share one index or when you need live cluster scaling. zvec wins when retrieval belongs to a single application and you'd rather ship a library than operate, secure and monitor one more server.

zvec vs Chroma

Both are easy to start with. Chroma optimises for prototyping speed; zvec brings a production engine, sparse vectors, full-text and filter fusion, and WAL durability out of the box, so there is no migration cliff when the project grows up.

zvec vs pgvector

If Postgres is already your source of truth and your scale is modest, pgvector is hard to beat for simplicity. We often pair a SQL source of truth with zvec precisely to keep heavy vector queries off the main database while keeping SQL canonical.

The pattern we deploy

One thing we never do is treat the vector index as the source of truth. In our deployments the SQL database keeps the canonical copy of every chunk: its text, its embedding, and a stamp of which model (and dimension) produced it. zvec is an acceleration layer that can be rebuilt from SQL at any time. Lose the volume, bump the schema, move to a platform without the native binding: search degrades gracefully and comes back. Nothing breaks, nothing is lost.

This split keeps the risky part boring. Upgrading the embedding model is a re-embed plus an index rebuild, not a migration. And because the index only ever returns candidate IDs that are re-read from SQL through the application's normal access-control path, the retrieval layer can't leak anything the caller wasn't allowed to see anyway.

zvec in practice

The API is what you'd hope for from an embedded database: a schema, a directory, and you're indexing (Python shown; SDKs also exist for Node.js, Go, Rust and Dart):

import zvec

schema = zvec.CollectionSchema(
    name="knowledge",
    vectors=zvec.VectorSchema("embedding", zvec.DataType.VECTOR_FP32, 1024),
)
collection = zvec.create_and_open(path="./index", schema=schema)

collection.insert([
    zvec.Doc(id="chunk_1", vectors={"embedding": doc_vector}),
])

results = collection.query(
    zvec.VectorQuery("embedding", vector=query_vector),
    topk=10,
)

That is the entire operational footprint. No docker-compose entry, no healthcheck, no connection pool: the "database" is a Python object over a directory.

Takeaways

• Embeddings are a commodity you can self-host. A 1 GB CPU container serves a multilingual state-of-the-art small model. If your data can't leave the building, it doesn't have to.
• Pick the model before the store. Retrieval quality lives in the embedding model; the store is an engineering choice about operations, not intelligence.
• Default to in-process, escalate to a server. Start with a library like zvec; adopt a Qdrant-class server the day several services genuinely need to share one index, not before.
• Keep SQL as the source of truth. A rebuildable index turns every scary migration into a routine re-index.

Florian, Founder & CTO at VBTech. We design and build AI-driven software solutions for B2B clients across the Middle East and Asia. If you need retrieval over data that can't leave your building, that is exactly the kind of system we build: get in touch.