Inside stage 2. Cross-encoder attention, ColBERT MaxSim, the learning-to-rank loss family (pointwise, RankNet, LambdaRank / LambdaMART), InfoNCE for first-stage training, MarginMSE distillation, and the formal derivations of DCG / NDCG / MRR / MAP.
The two-stage cascade itself — bi-encoder/cross-encoder asymmetry, indexability, ANN math, fusion derivations, latency & calibration — lives in RAG 08 — Two-Step Retrieval Architecture. The operator-facing overview with model tables, code snippets, and latency budgets is RAG 03 — Hybrid Search & Reranking.
Given a query q and a short-list D1 = {d1, …, dk1} from the first-stage retriever, a reranker is a function
that assigns a real-valued score to each (q, d) pair. The pipeline's stage-2 output is the top-k2 documents ordered by score:
D2 = argsortd ∈ D1↓ sθ(q, d) [: k2]Two implementation families dominate:
sθ(q,d) = hhead( BERTθ([CLS] q [SEP] d) )
One full transformer forward pass per (q,d) pair. Maximum quality, maximum cost.
sθ(q,d) = Σi maxj ⟨ Eqi, Edj ⟩
Per-token embeddings stored; scoring is a MaxSim reduction. Lower cost; precision close to a cross-encoder on most benchmarks.
This deck derives both, the loss functions that train them, the contrastive loss that trains the bi-encoder feeding stage 1, the distillation trick that lets the bi-encoder inherit the cross-encoder's quality, and the metrics that measure all of it.
A cross-encoder is the same transformer block you already know, applied to a concatenated input. The mathematics is unremarkable; the consequence is enormous.
Inside each layer, self-attention computes
Attn(H) = softmax( Q KT / √d ) V Q, K, V = HWQ, HWK, HWVwhere Q, K, V mix all rows of H — query tokens, document tokens, and special tokens. Concretely, the i-th row of Q KT contains the dot products between token i and every other token. If i is a query token and j is a document token, the entry (QKT)i,j is a direct, learned, soft-aligned similarity between them. The transformer then refines this in every subsequent layer.
A bi-encoder pools q and d into single vectors before they ever meet. By the time the dot product happens, all token-level structure has been crushed into a global summary. A cross-encoder keeps token-level structure all the way through and scores from a jointly-conditioned [CLS] embedding. This is why cross-encoders win on the BEIR average by 5–10 NDCG points despite using the same backbone architecture.
Per (q, d) pair, the dominant cost is self-attention over |q| + |d| tokens:
cost ≈ L · ((|q| + |d|)2 · D + (|q| + |d|) · D2) L = layers, D = hidden dimFor a 12-layer, 768-dim model with |q|+|d|=384, that is ~1–5 ms on an A100. Multiplied by k=100 candidates, the budget pressure that drives every other decision in this deck.
ColBERT (Khattab & Zaharia, SIGIR 2020) found a middle ground: keep per-token embeddings, but replace full cross-attention with a much cheaper interaction. The cost: a single, simple operator called MaxSim.
Let Eq ∈ ℝ|q| × D and Ed ∈ ℝ|d| × D be the L2-normalised token-level embeddings produced by the same BERT-style encoder. The score is:
Operationally: for each query token i, find the most similar document token j (over all j), take the similarity, and sum across query tokens.
The query and document are encoded independently (cheap, parallelisable, document side cacheable). They only meet at the final MaxSim, which is a closed-form algebraic operation — not another neural network. A bi-encoder interacts at a single scalar (the dot product of pooled vectors). A cross-encoder interacts continuously across all L layers. ColBERT interacts once, per token pair, at the end — hence “late”.
For each query token, MaxSim selects the single best-matching document token. This captures the most common signal in retrieval — the existence of a strong local alignment — without modelling the joint reasoning across query tokens. The empirical result: on BEIR-class corpora, ColBERTv2 (Santhanam et al., NAACL 2022) closes most of the gap between bi-encoder and cross-encoder quality at a fraction of the cost.
What it misses: conjunctive evidence requiring multiple query tokens to bind to coherent document regions. If a query is “X causes Y under condition Z”, MaxSim can match X, Y, Z to three different documents within the corpus by stitching together their best matches in different paragraphs. A cross-encoder can refuse that — its joint attention can spot that X, Y, Z need to co-occur.
For reranking a candidate set of k documents:
|q| × |d| dot-product matrix and a row-wise max–sum reduction. Trivially batched on GPU.End-to-end at k=200: typically 30–60 ms on a single A10, versus 150–300 ms for a comparable cross-encoder. PLAID (Santhanam et al., CIKM 2022) extends this with centroid-based pruning for first-stage use over the whole corpus, but the reranker case shown here needs none of that machinery.
Once the scoring function sθ is chosen, we have to train it. Learning-to-rank (LTR) is the body of methods that turns labelled (query, document, relevance) data into gradient signal. The classical taxonomy is three families, distinguished by what they treat as a training example.
Each (q, d) pair is one example. Predict the label (regression or classification) and minimise per-pair loss.
L = Σ ℓ(sθ(q,d), y)Simple, but ignores the fact that we only care about the relative order of documents per query, not absolute scores. Used in early TREC systems; rare today.
Each example is a pair of documents (d+, d−) for the same query, with d+ known to be more relevant. Loss penalises the order being wrong.
L = Σ ℓpair(s(q,d+), s(q,d−))The dominant family. RankNet (slide 05) and contrastive losses (slide 07) live here.
Each example is the entire ranked list for a query. Loss is computed over the whole list at once.
L = Σ ℓlist( s(q, ·), y(q, ·) )Best aligned with metrics like NDCG, but most metrics are non-differentiable. Workaround: LambdaRank (slide 06) — pairwise gradients weighted by metric impact.
What we actually want to optimise — NDCG@10, MRR, MAP — involves sort() and argmax, neither of which is differentiable. The pointwise family ignores this and hopes per-pair regression correlates with rank quality. The listwise family confronts it head-on (smoothed surrogates, listwise softmaxes). The pairwise family in its LambdaRank/LambdaMART form found the pragmatic answer: use a pairwise gradient, then weight it by how much swapping the two documents would change the metric. The next two slides spell that out.
RankNet (Burges et al., ICML 2005) is the original probabilistic pairwise ranking loss. Most modern reranker training — including cross-encoder fine-tuning — is RankNet or a close cousin.
For a query q and two documents di, dj, let si = sθ(q, di) and likewise sj. RankNet posits a sigmoid model for “di outranks dj”:
Pij = Pr[ rank(di) < rank(dj) ] = σ(si − sj) = 1 / (1 + e−(si − sj))Let P̄ij ∈ {0, ½, 1} be the target (1 if di is more relevant, 0 if less, ½ if tied). Cross-entropy loss:
In the typical “di strictly more relevant” case (P̄ij = 1), this collapses to:
Cij = log(1 + e−(si − sj)) softplus margin lossDifferentiate the cross-entropy form with respect to si:
∂Cij / ∂si = σ(si − sj) − P̄ij and ∂Cij / ∂sj = −(∂Cij / ∂si)Define
λij ≡ ∂Cij / ∂si = σ(si − sj) − P̄ijThen the total gradient on score si from a query's pair set is:
∂C / ∂si = Σ{j : (i,j) ∈ pairs} λij − Σ{j : (j,i) ∈ pairs} λjiThese λs are the “lambdas” that name LambdaRank. They are signed forces on each score: positive lambda ⇒ push score up; negative ⇒ push down.
From the gradient form: λij = σ(si − sj) − 1 when di should outrank dj. If the model already gets it right by a wide margin (si ≫ sj), σ≈1 and the gradient is near zero — nothing to learn. If the model gets it wrong (si < sj), the gradient is near −1 — full force. Therefore: training is gradient-dominated by the hard pairs — the close calls and the actual mistakes. This is why hard-negative mining (pick negatives that the current model scores near the positive) is the central trick in modern reranker training.
RankNet treats all pair-order errors as equivalent. NDCG does not. A swap of positions 1 and 2 hurts NDCG much more than a swap of positions 99 and 100. LambdaRank (Burges, Ragno & Le, NIPS 2006) fixes this by weighting each pairwise gradient by the metric impact of swapping that pair.
Take the RankNet lambda and multiply by the magnitude of the metric change if the two documents swapped positions in the current ranking:
λij(LR) = ( σ(si − sj) − P̄ij ) · | ΔNDCGij |where ΔNDCGij is the change in NDCG if documents i and j swap rank positions (all other documents fixed). This can be computed analytically from the rank positions and the labels:
ΔNDCGij = (Gi − Gj) · (1/Di − 1/Dj) / IDCGwhere Gi = 2reli − 1 (gain) and Di = log2(ranki + 1) (discount) — see slide 09 for the full DCG definition.
LambdaRank's gradient is not the gradient of any obvious loss function. Burges et al. introduced it as a heuristic: if we could differentiate NDCG, this is roughly what the gradient would look like. Later work (Donmez, Svore & Burges 2009; Wang et al. 2018) proved that LambdaRank gradients are in fact gradients of an underlying smooth surrogate for NDCG — the heuristic was a real algorithm in disguise. The practical result is the same: state-of-the-art on LETOR and TREC benchmarks for over a decade.
LambdaMART (Wu, Burges, Svore & Gao, 2010) takes the same lambdas and uses them as the residuals in gradient-boosted decision trees (MART = Multiple Additive Regression Trees):
LightGBM and XGBoost both ship a lambdarank objective that implements exactly this. For production rerankers operating over handcrafted features (BM25 score, dense cosine, click-through rate, freshness, recency …), LambdaMART remains the algorithm to beat. For end-to-end neural rerankers, the LambdaRank gradient idea is implemented inside SBERT-style training loops as MarginRankingLoss or with explicit listwise reweighting.
The bi-encoder that feeds stage 1 is also a ranker; it just has to be cheap. Modern dense retrievers (DPR, E5, BGE, GTE, Voyage) are trained with a contrastive loss called InfoNCE — a listwise softmax over one positive and many negatives.
For a batch of queries with positive documents and negatives, define the similarity
s(q, d) = ⟨ fθ(q), gφ(d) ⟩ / τ τ = temperature, often shared with φThe InfoNCE loss for a triple (q, d+, {d−k}) is:
L = − log es(q, d+) / ( es(q, d+) + Σk es(q, d−k) )This is exactly cross-entropy with the positive as the “correct class”: maximise the softmax probability that the model assigns to the positive document among (d+, d−1, …, d−N).
The temperature τ sharpens or softens the softmax. Small τ (e.g. 0.05) makes the loss focus aggressively on the hardest negative; large τ (e.g. 1) gives every negative similar weight. Tuning τ is one of the highest-leverage hyperparameters in contrastive retrieval — the trade-off is between learning sharp discriminative features (small τ) and stable training (larger τ).
The InfoNCE softmax over (d+, d−1, …, d−N) can be rewritten as a sum of pairwise terms involving s(q,d+) − s(q,d−k) — structurally the RankNet loss with one positive per group and N negatives. InfoNCE is, modulo normalisation, the listwise version of RankNet over a hand-picked candidate slate.
Cross-encoders are the gold standard for relevance but cannot run on the whole corpus. Bi-encoders can run on the whole corpus but are less accurate. Can the bi-encoder learn from the cross-encoder?
Yes. The trick is knowledge distillation with a loss tailored for rankers: MarginMSE (Hofstätter, Althammer, Schröder, Sertkan & Hanbury, arXiv 2010.02666, 2020).
Let sT be the teacher (cross-encoder) score and sS the student (bi-encoder) score. For a triple (q, d+, d−):
MT = sT(q, d+) − sT(q, d−) teacher margin MS = sS(q, d+) − sS(q, d−) student margin LMarginMSE = ( MS − MT )2Distilling the absolute scores would force the bi-encoder to match the cross-encoder's score magnitudes — impossible, because the two architectures have different output distributions. The margin — the gap between positive and negative — is the rank-determining quantity. Teaching the student to reproduce the teacher's margins teaches it to reproduce the teacher's rankings without forcing it to reproduce the teacher's score scale.
A cross-encoder trained on millions of (q, d+, d−) triples has implicitly learned the relevance manifold. Its margin MT on a fresh triple is a much richer label than the binary “positive vs negative” tag — it encodes the degree of relevance difference. Training the student on margins is closer to regression on a continuous teacher signal than to binary classification. Empirically: distilled bi-encoders match cross-encoder NDCG@10 to within 1–3 points on most benchmarks, at bi-encoder cost.
LMarginMSE, often combined with InfoNCE so the bi-encoder also gets contrastive signal: L = α·LInfoNCE + β·LMarginMSE.All the loss functions above are surrogates. The metrics that actually decide whether a system shipped are these four. Each is defined per query and averaged over the eval set.
For a ranked list of k documents with graded relevance labels rel1, …, relk:
DCG@k = Σi=1k (2reli − 1) / log2(i + 1) industry / TREC formulaTwo pieces of intuition:
DCG depends on the number of relevant documents and label distribution per query — raw DCG values are not comparable across queries. Fix: normalise by the DCG of the ideal ranking (relevant docs sorted by label, in order):
NDCG@k = DCG@k / IDCG@k ∈ [0, 1]NDCG@10 is the de facto BEIR-style benchmark metric for retrieval and reranking.
For each query, find the rank of the first relevant document; take its reciprocal; average over queries:
MRR = (1/|Q|) · Σq 1 / rankq(first relevant) 0 if none in top-kRight metric when there is one canonical answer per query (single-answer Q&A, fact lookup). Tells you nothing about ranking quality beyond the first relevant result.
For each query, compute Average Precision (AP):
AP@k = (1 / Rq) · Σi=1k Precision@i · [di relevant]where Rq is the total number of relevant documents for q and the indicator gates the sum to relevant positions. Then MAP = meanq APq. MAP uses binary relevance; rewards both finding many relevant docs and ranking them high.
NDCG@10 — default for everything; graded relevance, position-aware, comparable across queries. MRR — single-answer pipelines (factoid QA, codesearch, GitHub-issue retrieval). MAP — multi-document retrieval with binary labels; standard on TREC-style ad-hoc retrieval. Recall@k — the right metric for stage 1 of the cascade specifically (slide 05 of Deck 08).
Bringing it all together: the typical end-to-end training stack for a state-of-the-art open-weights retrieval pipeline (BGE, GTE, E5 lineage) uses every component above.
And what feeds each stage:
| Stage | Loss | Data shape | Output |
|---|---|---|---|
| Pretrain | MLM | Plain text | Backbone weights |
| Weak contrastive | InfoNCE, large batch in-batch negs | Query – passage pairs (e.g. Reddit, Stack Exchange) | Generic bi-encoder |
| Supervised contrastive | InfoNCE + mined hard negatives | MS MARCO / NQ + curated negs | Strong bi-encoder |
| Distillation | MarginMSE (+ InfoNCE) | Same triples, teacher scores | Distilled bi-encoder |
| Cross-encoder | RankNet / LambdaRank surrogate | (q, d+, d−) triples | Reranker |
The reranker (stage 5) becomes the teacher for the next iteration's distillation (stage 4). The improved bi-encoder (stage 4 output) becomes the source of harder hard negatives for the next iteration's contrastive training (stage 3) and for fine-tuning the next reranker (stage 5). This iterative loop is how BGE-v1 became BGE-M3 and how the GTE-Qwen and E5-mistral series scaled their quality without scaling their parameter counts.
For the operational side — latency budgets, model tables, cascade design — see RAG 03. For the architectural pillar this deck rests on (indexability, ANN math, the (k, quality, latency) Pareto frontier), see RAG 08. For ingestion (chunking, layout-aware parsing, deduplication), RAG 04.