Embedding Models — The Geometry of Meaning

Topics We'll Cover

• Why Embeddings — The Geometry of Meaning
• Bi-Encoder vs Cross-Encoder
• The Modern Model Families
• ColBERT & Late Interaction
• Matryoshka Embeddings — One Model, Many Dimensions
• Domain Adaptation — TSDAE, GPL, Hard Negative Mining
• Evaluating Embeddings — MTEB, BEIR, In-Domain Evals
• Practical Sizing — Dim, Throughput, GPU vs CPU
• What to Take Away

Why Embeddings — The Geometry of Meaning

An embedding is a real-valued vector that encodes the semantic position of a token sequence in a high-dimensional space. The key property is metric: similar meanings live close together under cosine similarity or dot product, enabling sub-linear nearest-neighbour search instead of full corpus comparison.

There are two competing traditions:

Bi-Encoder vs Cross-Encoder

The most important architectural fork. Nearly every production RAG pipeline uses both: bi-encoder for retrieval, cross-encoder for reranking.

The Modern Model Families

As of early 2025, the landscape has consolidated around a handful of open and proprietary families. Knowing the lineage matters: a model trained with contrastive InfoNCE loss behaves very differently from one trained with MNRL (Multiple Negatives Ranking Loss).

ColBERT & Late Interaction

ColBERT (Contextualised Late Interaction over BERT, Khattab & Zaharia 2020) occupies a middle ground: it is cheaper than a cross-encoder and more expressive than a bi-encoder. The insight is to defer the query–document interaction until after both sequences are encoded, but to keep per-token vectors rather than collapsing to a single pool.

The MaxSim operator: for each query token vector q𝑖, find the maximum dot-product over all document token vectors dⱼ. Sum the per-token maxima. This yields a score that respects the term-level alignment between query and document without requiring all tokens to interact in a single attention matrix.

import numpy as np

def maxsim_score(Q: np.ndarray, D: np.ndarray) -> float:
    # Q: [n_q, 128]  D: [n_d, 128], both L2-normalised
    scores = Q @ D.T          # [n_q, n_d]
    return float(scores.max(axis=1).sum())

# Index storage: one 128-dim vector per token per document
# ColBERTv2 + PLAID compresses to ~4 bits/dim via residual quantisation

Matryoshka Embeddings — One Model, Many Dimensions

Matryoshka Representation Learning (MRL, Kusupati et al. 2022) trains a single encoder such that any prefix of the output vector is itself a valid, useful embedding. You can truncate a 768-dim vector to 64 dims and still get reasonable retrieval, with quality scaling smoothly as you increase dimension.

Domain Adaptation — TSDAE, GPL, Hard Negative Mining

Off-the-shelf models trained on MS MARCO or NLI datasets often degrade significantly on specialised domains (legal, biomedical, financial, customer support). Three techniques address this without requiring manual annotation at scale.

from sentence_transformers import SentenceTransformer, losses
from sentence_transformers.datasets import NoDuplicatesDataLoader

model = SentenceTransformer("BAAI/bge-base-en-v1.5")

# GPL-generated triplets: (query, positive_passage, negative_passage)
train_loss = losses.MultipleNegativesRankingLoss(model)

# warm-up 10 % of steps, then cosine LR decay
model.fit(
    train_objectives=[(dataloader, train_loss)],
    epochs=1,
    warmup_steps=100,
    output_path="./domain-adapted-bge",
)

Evaluating Embeddings — MTEB, BEIR, In-Domain Evals

Never trust a model ranking from a provider's blog post. Understand what each benchmark measures before picking a model for production.

Practical Sizing — Dim, Throughput, GPU vs CPU

Embedding infrastructure is often the largest ongoing compute cost in a RAG system. The decisions compound: higher dimension → larger index → slower ANN → more RAM → higher hosting cost.

* batch size 64, max 512 tokens, fp16 where supported. Approximate; varies with sequence length.

What to Take Away

• Dense beats sparse for paraphrase retrieval; sparse beats dense for exact-term recall. Hybrid is almost always better than either alone.
• Bi-encoder for retrieval, cross-encoder for reranking. Never use a cross-encoder at index time; never use a bi-encoder as the final ranker if quality matters.
• BGE-M3 is the strongest open-weights general-purpose choice as of 2025-Q1. It supports dense, sparse, and ColBERT modes from a single model.
• ColBERT + PLAID is the right architecture when you want cross-encoder quality at bi-encoder latency and can afford 4–8× the storage.
• Matryoshka (MRL) lets you trade quality for speed at deployment time without retraining. OpenAI text-embedding-3 and nomic-embed-text-v1.5 both support it.
• Domain-adapt before you scale. GPL on 10 k domain sentences will outperform the best off-the-shelf model on your corpus if your domain is specialised.
• Measure in-domain Recall@100 — MTEB rankings are a starting point, not a verdict.