RAG Deep Dive: Search Types, Embedding Models, and Choosing the Right Vector Store

Retrieval-Augmented Generation (RAG) sounds simple on the surface — fetch relevant documents, pass them to an LLM, get a grounded answer. But the moment you move beyond a basic prototype, you realize there are a dozen decisions hiding inside that one sentence. Which retrieval strategy fits your data? Which embedding model should you use? Should the model you use to store vectors be the same one you use to retrieve them? And which vector database won't become a bottleneck at scale?

This post breaks all of that down, with enough depth to actually inform architectural decisions.

What is RAG?

Large Language Models (LLMs) are powerful — but they have a fundamental limitation: their knowledge is frozen at their training cutoff. They don't know what happened last week, they don't know your company's internal documents, and they can't look up a customer's order history. When they don't know something, they often make something up — this is called hallucination.

RAG is the solution to this problem. Instead of relying purely on what the model memorized during training, RAG gives the model access to an external knowledge source at query time. The flow looks like this:

User Query
↓
Retrieve relevant documents from a knowledge base
↓
Inject those documents into the LLM's prompt as context
↓
LLM generates an answer grounded in the retrieved content

Think of it like an open-book exam. The LLM is the student — instead of answering from memory, it reads the relevant pages before writing the answer. The quality of the answer depends heavily on which pages get retrieved.

The Two Phases of RAG

Phase 1 — Indexing (offline): Your source documents (PDFs, web pages, internal wikis, databases) are split into chunks, converted into vector embeddings using an embedding model, and stored in a vector database alongside the original text.

Phase 2 — Retrieval + Generation (online, at query time): The user's query is embedded using the same model, the vector database finds the closest matching chunks, and those chunks are stuffed into the LLM's prompt as context before the model generates its response.

[Source Docs] → [Chunking] → [Embedding Model] → [Vector DB]
↑
[Query Embedding]
↓
[Top-K Chunks]
↓
[LLM Prompt = Query + Chunks]
↓
[Final Answer]

Why Not Just Use Fine-Tuning?

A common question is: why not just fine-tune the LLM on your private data instead?

Fine-tuning bakes knowledge into model weights — it's expensive, slow to update, and you lose source traceability (you can't easily cite where an answer came from). RAG keeps your knowledge base live and updatable — add a new document and it's immediately retrievable without touching the model. For most production use cases, RAG is the right answer. Fine-tuning is complementary for style and behavior, not for injecting knowledge.

Types of RAG Searches

The retrieval layer is where most RAG systems either succeed or quietly fail. Choosing the wrong strategy means your LLM gets the wrong context — and a confident wrong answer is worse than no answer.

1. Dense (Semantic) Retrieval

Dense retrieval converts both your query and your documents into high-dimensional vectors using an embedding model. Similarity is measured by cosine distance or dot product. It excels at understanding meaning, not just keywords.

How it works: Both documents and the query live in the same vector space. Similar meaning → closer vectors.

Example: Query: "how do I reset my account password?" — retrieves a document titled "Account Recovery Options" even if the word "reset" never appears in it.

Best for: Conversational search, semantic Q&A, knowledge bases where phrasing varies.

Limitation: Dense models can miss exact terms — if a user searches for error code 4031, a semantic model might retrieve loosely related error content instead of the exact match.

2. Sparse (Keyword) Retrieval — BM25 / TF-IDF

Sparse retrieval is the older approach, built on term frequency. BM25 is the gold standard here. It represents documents as sparse vectors where each dimension maps to a vocabulary token, and it ranks documents based on token overlap and frequency.

Example: Query: "invoice PDF download 2024" — BM25 will surface documents that contain these exact terms, which dense retrieval might dilute by generalizing.

Best for: Legal documents, code search, product catalogs, financial data — anywhere exact terms matter.

Limitation: It has zero semantic understanding. A synonym or paraphrase will score 0 if the token isn't present.

3. Hybrid Search

Hybrid search combines dense and sparse retrieval, then merges the result sets using a ranking algorithm. The most common approach is Reciprocal Rank Fusion (RRF) — a simple, parameter-free way to blend ranked lists from both retrievers.

Some implementations use a weighted sum with a tunable alpha parameter that controls how much to lean semantic vs. keyword.

Why it matters: Real-world queries are neither purely semantic nor purely lexical. Hybrid search is consistently the top performer across most retrieval benchmarks (BEIR, MTEB).

Use case: Enterprise search, e-commerce product discovery, technical documentation search.

4. HyDE — Hypothetical Document Embeddings

HyDE is a clever trick: instead of embedding the raw user query, you ask an LLM to generate a hypothetical answer to the query first, then embed that. The intuition is that a generated answer lives closer in the embedding space to actual answers in your corpus than the query itself does.

Example: Query: "What are the tax implications of converting an LLC to an S-Corp?"
HyDE generates a paragraph-length hypothetical answer and embeds it — pulling in far richer signal than the 12-word query alone.

Best for: Complex, open-ended questions where query-to-document embedding distance is naturally large.

Tradeoff: Adds an extra LLM call per query, which increases latency and cost.

5. Multi-Vector / ColBERT-style Retrieval

Instead of encoding a document into a single vector, ColBERT encodes each token of a document into its own vector. At retrieval time, it performs a late interaction — comparing query token vectors against document token vectors using a MaxSim operation.

This is expensive to store but dramatically more precise, especially for long documents where a single embedding collapses too much information.

Use case: Long-form document retrieval (legal contracts, research papers), scenarios where precision matters more than speed.

6. Graph RAG

Graph RAG structures your knowledge as a graph — entities as nodes, relationships as edges — and traverses connections rather than doing pure vector similarity. When a query asks about relationships between entities, graph traversal retrieves context that flat vector search would miss entirely.

Example: "Which engineers have worked on projects that use the same vendor as the Athens contract?" — this requires relational hops that a single embedding can't encode.

Best for: Knowledge graphs, organizational data, research networks, anything with rich entity relationships.

Embedding Models: What Goes Into Your Vector DB

The embedding model is arguably the most critical choice in a RAG system. It determines the quality of your vector space — and therefore the ceiling on your retrieval accuracy.

Popular Embedding Models

Model. Dimensions Notes

text-embedding-3-small (OpenAI) 1536 (truncatable) Best cost/performance ratio, multilingual

text-embedding-3-large (OpenAI) 3072 Highest accuracy in the OpenAI family

text-embedding-ada-002 (OpenAI) 1536 Legacy, still widely deployed

embed-english-v3.0 (Cohere) 1024. Strong performance, built-in int8 quantization

all-MiniLM-L6-v2

(Sentence Transformers). 384 Fast, lightweight, open-source, self-hostable

bge-large-en-v1.5 (BAAI). 1024 Top open-source performer on MTEB leaderboard

nomic-embed-text-v1.5 (Nomic). 768 Open-source, supports Matryoshka (variable dims)

jina-embeddings-v3 1024 Multilingual, 8192 token context

Dimensions Matter — But More Isn't Always Better

Higher dimensions capture more nuance but cost more in storage and compute. OpenAI's text-embedding-3 models support Matryoshka Representation Learning — you can truncate the vector to a smaller dimension (e.g., 256 or 512) and still get strong performance with lower storage overhead. This is a useful optimization when you're storing tens of millions of vectors.

Domain-Specific Models

General embedding models are trained on web-scale text. If your corpus is medical, legal, or code-heavy, a domain-specific model will outperform a general one significantly:

• Code: voyage-code-2, CodeBERT
• Legal: legal-bert-base-uncased, Cohere's legal fine-tunes
• Medical/Bio: PubMedBERT, BioLinkBERT
• Multilingual: multilingual-e5-large, paraphrase-multilingual-mpnet-base-v2

Should You Use the Same Model for Storing and Retrieving?

Yes. Absolutely. This is non-negotiable.

Here's why: an embedding model maps text to a specific point in a specific vector space. That space is defined by the model's weights. If you embed your documents with Model A and then embed a query with Model B, the resulting vectors exist in completely different geometric spaces. Distance calculations between them are meaningless — you'll get random, garbage results.

This is called embedding space mismatch, and it's a silent failure — no error is thrown, results just quietly become irrelevant.

What Happens When You Need to Switch Models?

This is a real operational challenge. You might need to switch because:

• A better model is released (and you want the quality gain)
• Your current model vendor raises prices or deprecates a version (OpenAI deprecated ada-001)
• You're moving from a cloud API to a self-hosted model for cost/privacy reasons
• Your data domain has shifted and you need a fine-tuned model

The answer: full re-embedding. You must re-embed every document in your corpus with the new model, rebuild your index, and only then switch query traffic to use the new model. There's no partial path.

This has real implications:

• Version pin your model IDs — store which model version was used for each chunk alongside the vector.
• Keep the old index live during transition if you need zero-downtime switching.
• Budget for re-embedding compute — for millions of documents, this can be hours of API calls or GPU time.

Some teams hedge this by storing the raw text chunks alongside vectors, making re-embedding a batch job rather than requiring re-ingestion of original source data.

Choosing a Vector Store

The vector database is where your embeddings live, and your choice affects latency, scalability, filtering capabilities, and operational complexity.

pgvector — When You Already Have PostgreSQL

pgvector is a PostgreSQL extension that adds a vector data type and approximate nearest neighbor (ANN) search via HNSW and IVFFlat indexes.

Why pgvector is often the right first choice:

• Zero new infrastructure if you're already on Postgres
• Full SQL — filter by metadata, join to user tables, all in one query
• ACID transactions, familiar backups, existing auth

sql

-- Create a table with a vector column
CREATE TABLE documents (
id SERIAL PRIMARY KEY,
content TEXT,
embedding vector(1536),
metadata JSONB
);

-- Query: find 5 most similar documents to a given vector
SELECT content, 1 - (embedding <=> '[0.12, 0.34, ...]'::vector) AS similarity
FROM documents
ORDER BY embedding <=> '[0.12, 0.34, ...]'::vector
LIMIT 5;

Limitation: At very large scale (tens of millions of vectors), dedicated vector databases will outperform pgvector in throughput and index build time.

Pinecone — Fully Managed, Production-Ready

Pinecone is a purpose-built managed vector database. No infra to manage, automatic scaling, and it supports namespaces for multi-tenancy out of the box.

Best for: Teams that want to ship fast without managing vector infra. It handles billions of vectors gracefully.

Tradeoff: Vendor lock-in and cost at high query volumes.

Qdrant — Open-Source, High Performance

Qdrant is written in Rust, has excellent benchmark performance, and supports payload-based filtering (filtering by metadata before the ANN search, not after — a huge accuracy advantage). It can be self-hosted or used as a managed cloud service.

Standout feature: Sparse-dense hybrid search is natively supported — you can do BM25 + dense retrieval in a single query.

Weaviate — Schema-First, GraphQL API

Weaviate uses a schema-based approach and exposes a GraphQL interface. It has native support for BM25 + semantic hybrid search and integrates directly with embedding model providers (you can configure it to auto-embed on insert).

Best for: Teams that want a structured data model and don't want to manage embedding separately.

Chroma — Lightweight, Great for Development

Chroma is an in-process vector store that runs embedded in your Python application. It's the fastest way to prototype a RAG system — no server needed.

python

Not for production at scale — it's a dev/small-project tool.

FAISS — When You're Building Your Own

Facebook's FAISS is a library, not a database. It gives you maximum control over ANN indexing algorithms (HNSW, IVF, PQ) with no overhead. Used inside systems like Pinecone and Qdrant under the hood.

Best for: ML engineers building custom retrieval pipelines who want raw index control.

Putting It All Together

A well-designed RAG system is a set of deliberate choices, not defaults. A practical starting point for most teams:

• Search strategy: Start with hybrid search (dense + BM25 via RRF). Add HyDE if query quality is poor. Move to ColBERT if precision on long documents matters.
• Embedding model: text-embedding-3-small or bge-large-en-v1.5 for general use. Domain-specific fine-tune if your corpus is specialized. Always store the model name + version alongside every vector.
• Same model in, same model out — lock this in from day one. Build a re-embedding pipeline before you need it.
• Vector store: pgvector if you're already on Postgres and under 10M vectors. Qdrant or Weaviate for self-hosted at scale. Pinecone if you want zero infra management.

The retrieval layer is not where you want to cut corners — it directly determines what context your LLM sees. Get it right and the LLM's job becomes dramatically easier.