CLIP & Contrastive Vision-Language Learning

Topics We’ll Cover

• Multimodal Alignment — Text + Image in One Space
• CLIP Architecture & Training
• Contrastive Loss Math — InfoNCE
• OpenCLIP, EVA-CLIP, SigLIP — What Each Fixed
• Sigmoid vs Softmax Loss
• Scale Curves — Dataset, Model, Batch Size
• Zero-Shot Classification & Retrieval
• As a Building Block (Text-to-Image, VLMs)
• What to Take Away

Multimodal Alignment — Text + Image in One Space

Before CLIP, vision models were trained on fixed label sets. To add a new category you re-trained the classification head. CLIP (Contrastive Language-Image Pre-training, Radford et al., OpenAI 2021) learns a joint embedding space from 400 million noisy image-text pairs scraped from the web. Both modalities collapse to a common unit-normed vector; cosine similarity becomes the universal comparator.

The key insight is that the web already contains supervisory signal: alt-text, captions, surrounding context. The model does not need human-curated labels — it just has to learn to assign high similarity to the correct (image, text) pair and low similarity to the 4,095 other texts in a batch of 32,768.

CLIP Architecture & Training

CLIP uses two independent encoders that share no weights. After encoding, both outputs are L2-normalised and projected to the same dimensionality. Training maximises the cosine similarity of correct pairs and minimises it for all other pairs within the batch.

Contrastive Loss Math — InfoNCE

The training objective is the InfoNCE loss (van den Oord et al., 2018), also called NT-Xent in SimCLR. CLIP applies it symmetrically to both modalities. For a batch of N pairs, the model sees an N × N similarity matrix; diagonal entries are the positives.

# I = image embeddings  [N, d],  L2-normalised
# T = text  embeddings  [N, d],  L2-normalised
# logit_scale = exp(log_temperature)  (learnable, init ~14 = 1/0.07)

import torch, torch.nn.functional as F

logits = logit_scale * I @ T.T          # [N, N]
labels = torch.arange(N, device=logits.device)

loss_i = F.cross_entropy(logits,   labels)  # image→text
loss_t = F.cross_entropy(logits.T, labels)  # text→image
loss   = (loss_i + loss_t) / 2

The temperature τ controls peakiness of the softmax distribution. CLIP initialises log(τ) = log(1/0.07) ≈ 2.66 and learns it jointly. A lower τ sharpens the distribution and amplifies gradients from hard negatives; too low and the loss saturates on easy pairs and ignores hard ones.

Why cross-entropy implements InfoNCE

Cross-entropy over a row of the logit matrix is equivalent to maximising the log-probability of the positive pair relative to all negatives — which is exactly the InfoNCE bound on mutual information I(I;T). With batch size N, the bound is at most log(N) bits. Larger batches give a tighter bound and more hard negatives per update.

OpenCLIP, EVA-CLIP, SigLIP — What Each Fixed

CLIP's closed training data created a reproducibility gap. Three major follow-ups each targeted a different limitation:

Sigmoid vs Softmax Loss

The original CLIP uses softmax across the entire batch for each positive, making the problem a single-correct multi-class classification. SigLIP treats each pair in the N × N matrix as an independent binary classification using sigmoid binary cross-entropy. This has deep consequences.

# logits: [N, N] = image @ text.T * scale
# labels: +1 on diagonal, -1 off-diagonal

import torch

labels = 2 * torch.eye(N, device=logits.device) - 1   # {-1, +1}

# Per-pair sigmoid binary cross-entropy:
loss = -torch.nn.functional.logsigmoid(labels * logits).sum() / N

# Equivalently (numerically stable):
# loss = F.binary_cross_entropy_with_logits(logits, (labels+1)/2)

Key differences

Scale Curves — Dataset, Model, Batch Size

The DataComp benchmark (Gadre et al., 2023) and a series of OpenCLIP ablations produced the clearest picture yet of what actually drives CLIP zero-shot performance. The answer is mostly data quality, then model size, then compute budget — a different ranking from language-model scaling laws.

The key takeaways from DataComp

• DataComp-1B vs LAION-2B: same ViT-L/14 model, half the data, but 2.3 pp higher IN-1K accuracy — curation via CLIP-score filtering plus image-text alignment scoring outweighs raw scale.
• Batch size: OpenCLIP ablations show diminishing returns beyond B = 32,768 for ViT-L. The benefit of larger batches (more hard negatives) plateaus when the model saturates easy pairs.
• Model size: ViT-bigG (1.84B image params) improves over ViT-H (632M) by ~2 pp — significant but expensive. EVA MIM pre-training closes most of the gap at lower cost.

Zero-Shot Classification & Retrieval

CLIP's canonical downstream use is zero-shot classification: construct one text prompt per class label, encode all prompts, then rank the image against the text embeddings by cosine similarity. No fine-tuning, no labelled data.

import open_clip, torch
from PIL import Image

model, _, preprocess = open_clip.create_model_and_transforms(
    'ViT-L-14', pretrained='datacomp_xl_s13b_b90k'
)
tokenizer = open_clip.get_tokenizer('ViT-L-14')

# Build class embeddings once (cache for large label sets)
classnames = ["a dog", "a cat", "a car"]
templates  = ["a photo of {}", "a {} in the wild"]
texts = [t.format(c) for c in classnames for t in templates]
with torch.no_grad():
    txt_feats = model.encode_text(tokenizer(texts)).float()
    txt_feats /= txt_feats.norm(dim=-1, keepdim=True)
    # Average over templates for each class
    txt_feats = txt_feats.view(len(classnames), len(templates), -1).mean(1)
    txt_feats /= txt_feats.norm(dim=-1, keepdim=True)

img = preprocess(Image.open("photo.jpg")).unsqueeze(0)
with torch.no_grad():
    img_feat = model.encode_image(img).float()
    img_feat /= img_feat.norm(dim=-1, keepdim=True)

probs = (img_feat @ txt_feats.T).softmax(dim=-1)
pred  = classnames[probs.argmax()]

Prompt engineering matters

OpenAI found that ensembling 80 hand-crafted templates (“a photo of a {}”, “a blurry photo of the {}”, …) improved IN-1K from 71.3% to 76.2% for ViT-L/14. The averaged embedding is more robust than any single prompt. This gave rise to the field of CLIP prompt tuning (CoOp, CoCoOp, CLIP-Adapter) which learns the prompt prefix in continuous embedding space rather than by hand.

As a Building Block (Text-to-Image, VLMs)

CLIP is rarely deployed standalone. Its embedding space and image encoder are load-bearing components in three major downstream architectures:

The frozen-vs-tuned decision

Most text-to-image systems keep the CLIP text encoder frozen during diffusion training, treating it as a fixed feature extractor. LLaVA-1.5 keeps the CLIP image encoder frozen too (only the MLP projection and LLM are trained). This is why a fine-tuned diffusion model that breaks the CLIP conditioning — e.g. by further training the text encoder on NSFW data — loses compositionality on common prompts.

What to Take Away

• CLIP collapses two modalities into one cosine-comparable space using noisy web-scale image-text pairs and InfoNCE loss. No manual labels required.
• InfoNCE is symmetric cross-entropy on an N×N batch similarity matrix. Temperature is learnable but must be clipped to prevent gradient explosion.
• OpenCLIP gave us reproducibility (open data + code + weights). EVA-CLIP unlocked huge ViTs via MIM warm-start. SigLIP replaced the batch-softmax with per-pair sigmoid BCE, removing the false-negative problem and enabling smaller batch training.
• SigLIP-SO400M/14 is the current best-practice image encoder for downstream VLMs: 83.1% IN-1K, strong OCR & chart transfer, open weights on HuggingFace.
• Data quality beats data quantity. DataComp-1B (1B curated pairs) outperforms LAION-2B (2B raw pairs) with the same model and compute.
• CLIP is plumbing. It powers DALL-E 2 priors, Stable Diffusion conditioning, LLaVA projection layers, retrieval augmentation, and CLIP-score evaluation metrics.