Learning Without a Teacher

Every method you've seen so far has needed labels. Detection needs annotated bounding boxes. Segmentation needs pixel-level masks. Even ImageNet classification needs a human to write "this is a golden retriever." That labelling is expensive — a single ImageNet costs millions of dollars in human time, and there are only so many golden-retriever-tagged photos in the world.

Then somebody asked: what if the supervision could come from the data itself?

You hand the network an image. You don't tell it what the image is. But you create a task that the image can answer with its own structure — "predict what colour this grayscale patch should be", "unscramble these jigsaw pieces", "figure out which two crops came from the same image". The network learns features by solving these synthetic puzzles — and those features turn out to be useful for downstream tasks.

This is self-supervised learning (SSL), and over five years it has become the dominant pretraining recipe in vision. Every modern model — CLIP, DINO, MAE, the vision encoders inside PaliGemma and Qwen2-VL — is SSL-pretrained. No labels. Just structure.

Where the supervision comes from

The lecture opens by listing the structural signals SSL exploits.

In language: grammar (predict the next word), fill-in-the-blanks (BERT), sentence ordering.

In images — the pretext-task era:

• Colorisation — predict the colour version of a grayscale image.
• Jigsaw puzzles — given 9 shuffled patches in a $3\times 3$ grid, predict the original arrangement.
• Neighbourhood proximity — given two patches, predict their spatial relationship.

These were clever, but the features they produced were only OK. The link between "solving jigsaws" and "classifying golden retrievers" was indirect.

What replaced them is a more direct approach: contrastive learning.

The taxonomy your lecture gives you

The lecture explicitly classifies vision SSL into four families. Memorise this:

1. Old-school SSL — jigsaw, colorisation, autoencoders. 2. Contrastive — SimCLR, MoCo, DINO. 3. Language-image contrastive — CLIP, SigLIP. 4. Generative — masked autoencoders (MAE).

This unit is about families 2 and 3.

The Gelato Bet

Alyosha Efros's Gelato Bet: that by a certain deadline, a single self-supervised model would match supervised ImageNet pretraining on a comprehensive benchmark. The bet was won — SSL caught up around 2020–2021. Vision-research folklore worth knowing.

The contrastive recipe

All contrastive methods rest on one principle:

For each image, generate two augmented "views." The two views are a positive pair (they came from the same image). All other images in the batch are negatives. Train the network so positive pairs have similar embeddings, and positive-vs-negative pairs have different embeddings.

That's it. What changes between methods is what augmentations, where the negatives come from, and how the loss is structured.

SimCLR — the simplest contrastive framework

SimCLR (Chen et al., ICML 2020) is the cleanest demonstration of the recipe. "Simple framework for Contrastive Learning of visual Representations." The title is honest.

Four components — name them on the exam

1. Augmentation pipeline — generates positive pairs. 2. **Encoder $f_{\theta }$** — a ResNet — produces representations $h$. 3. **Projection head $g_{\varphi }$** — a 2-layer MLP — maps $h$ to a contrastive space $z$. **The loss is on $z$, not $h$. After pretraining, $g_{\varphi }$ is discarded** and only $h$ is used downstream. 4. NT-Xent / InfoNCE loss.

Aggressive augmentations

SimCLR's headline finding: aggressive augmentation is essential. Random crop + colour jitter is the dominant pair; both must be strong, and the combination matters more than either individually. Crops teach scale/translation invariance, colour jitter teaches colour invariance, together they prevent the network from shortcutting on easy cues.

NT-Xent (= InfoNCE)

Given a batch of $N$ images, after augmentation you have $2N$ views. For one positive pair $(z_i,z_j)$:

with $\text{sim}(u,v)=u^{\top }v/(\Vert u\Vert \Vert v\Vert )$ and $\tau \in [0.07,0.5]$.

The connection — exam gold: NT-Xent is softmax cross-entropy. The "logits" are similarity scores; the "true label" is the index of the positive partner. Same equation, two names.

What SimCLR taught the field

• Bigger batches are dramatically better — the denominator covers more negatives. SimCLR scaled to batches of 4096–8192.
• The projection head matters — putting the loss on $g_{\varphi }(h)$ rather than directly on $h$ gives 5–10% better downstream accuracy.
• Aggressive augmentations are essential.

But batch 8192 means SimCLR only runs on TPU pods. The next paper solved that.

MoCo — Momentum Contrast

MoCo (He et al., CVPR 2020) keeps the contrastive recipe but decouples the number of negatives from the batch size.

Two key ideas

Idea 1 — a memory queue. Maintain a large FIFO queue of past representations ($K=65,536$). When you process a new batch, the negatives come from this queue, not the current batch. Huge queue → many negatives → strong learning signal, without needing a huge batch.

Idea 2 — a momentum encoder. Maintain a separate slowly-updated encoder $f_{\xi }$ that is an EMA of the online encoder $f_{\theta }$:

The momentum encoder produces all keys (positives + the negatives put into the queue). Because $\xi$ changes slowly, the queue stays consistent — all entries look like they came from approximately the same encoder.

The loop

For each batch: query $q=f_{\theta }(x_{\text{aug1}})$; positive key $k^{+}=f_{\xi }(x_{\text{aug2}})$; negative keys = the queue. InfoNCE: $q$ tries to match $k^{+}$ against $\{k^{-}\in \text{queue}\}$. Backprop only updates $\theta$. Update $\xi$ by EMA of $\theta$. Enqueue current batch's keys, dequeue oldest.

Trade-off to memorise: SimCLR needs huge batches because every batch is the negative pool. MoCo decouples — small batches, large queue.

DINO's teacher EMA (next unit, ${\theta }_t\leftarrow \lambda {\theta }_t+(1-\lambda ){\theta }_s$) is directly borrowed from MoCo's momentum encoder idea.

BYOL — no negatives at all

BYOL (Grill et al., NeurIPS 2020) — "Bootstrap Your Own Latent" — asked the most heretical question of the contrastive era: what if we just dropped the negatives entirely?

The contrastive intuition was: without negatives, the network will collapse — it'll output the same vector for every image. BYOL showed this isn't true if the architecture is right.

The setup

Online network: $f_{\theta }\to g_{\theta }\to q_{\theta }$ — encoder + projector + predictor.

Target network: $f_{\xi }\to g_{\xi }$ — encoder + projector (no predictor). $\xi \leftarrow$ EMA of $\theta$.

Loss: $\Vert \overline{q_{\theta }(g_{\theta }(f_{\theta }(v)))}-\overline{\text{sg}[g_{\xi }(f_{\xi }(v^{\prime }))]}{\Vert }^2=2-2\cos (\text{online},\text{target})$.

Three key architectural elements:

1. **Predictor $q_{\theta }$** — an extra MLP on the online branch only. Asymmetric. Critical. 2. Stop-gradient on the target branch — no backprop through $\xi$. 3. **EMA update of $\xi$ from $\theta$** — same idea as MoCo.

No negatives anywhere.

Why doesn't it collapse?

This is one of the most-discussed papers of 2020. The intuition:

• The predictor breaks symmetry — only the online side can perfectly match, so the system can't trivially converge to a constant.
• The EMA delay means the target is always slightly behind the online — chasing a moving target prevents lock-up.

Empirically, it just works. BYOL matched SimCLR on ImageNet without using a single negative. This was a turning point that led directly to DINO.

SwAV — contrastive clustering

SwAV (Caron et al., NeurIPS 2020) — "Swapping Assignments between Views" — takes yet another approach: online clustering.

Learn $K$ prototype vectors $\{c_1,\dots ,c_K\}$ — a small codebook (e.g. $K=3000$).

For each augmented view $v$:

1. Compute $z=f_{\theta }(v)/\Vert f_{\theta }(v)\Vert$. 2. Compute a soft cluster assignment $q\in {\mathbb{R}}^K$ via Sinkhorn-Knopp — enforces that across a batch each prototype is used approximately equally (prevents collapse to one prototype). 3. The swap: predict $q_{v^{\prime }}$ (the other view's cluster assignment) from $z_v$.

Two views of the same image should have the same cluster assignment. SwAV enforces this without ever directly comparing pairs of images — just comparing each view's features to the shared prototypes.

The family in one slide

| Method | Year | Negatives? | Key innovation | | --- | --- | --- | --- | | SimCLR | 2020 | Yes (in-batch) | Aggressive aug + projection head | | MoCo | 2020 | Yes (queue) | Memory bank + momentum encoder | | BYOL | 2020 | No | Predictor + EMA + stop-grad | | SwAV | 2020 | Implicit (clusters) | Online clustering + Sinkhorn | | DINO | 2021 | No | Self-distillation + multi-crop |

CLIP — when language joined the party

Everything above used images only — augmentations of the same image as the source of supervision. CLIP (Radford et al., 2021) does something different: it uses paired text as the supervision signal.

Why language matters as supervision

Three limitations of category-label supervision:

1. Hard to scale. ImageNet has 22k classes. Beyond ~10k, getting good per-class labels is infeasible. 2. Limited descriptive potential. "White shirt" vs "blue shirt" — separate classes? You'd need 10k just for clothing colours. 3. Not compositional. "Laptop on top of a table" combines two objects and a spatial relationship.

Natural language has none of these limitations, and the internet has billions of image-text pairs for free.

A piece of history — DeViSE (2013)

DeViSE (Frome et al., NeurIPS 2013) — Deep Visual-Semantic Embedding — was the earliest attempt to learn a joint visual-textual embedding space. CLIP came 8 years later and made the idea work at scale.

The CLIP setup

Two encoders, one shared embedding space:

**Image encoder $f_I$** — ResNet or ViT — outputs $I\in {\mathbb{R}}^d$.

**Text encoder $f_T$** — Transformer — outputs $T\in {\mathbb{R}}^d$.

Both L2-normalised; both projected to a shared $d_e$-dim space.

The pseudocode (memorise)

`` I_f = image_encoder(I) # [n, d_i] T_f = text_encoder(T) # [n, d_t] I_e = l2_normalize(I_f @ W_i, axis=1) # [n, d_e] T_e = l2_normalize(T_f @ W_t, axis=1) # [n, d_e] logits = (I_e @ T_e.T) * exp(t) # [n, n] pairwise similarities labels = arange(n) # diagonal is positive loss_i = cross_entropy(logits, labels, axis=0) # image-to-text loss_t = cross_entropy(logits, labels, axis=1) # text-to-image loss = (loss_i + loss_t) / 2 # symmetric ``

It's symmetric InfoNCE in matrix form. The $n\times n$ logit matrix has:

• Diagonal entries = positive pairs (image $i$ ↔ text $i$).
• Off-diagonal = negative pairs.
• Cross-entropy along rows = image-to-text retrieval. Along columns = text-to-image.

SigLIP later replaced this softmax with independent sigmoids; CLIP is the softmax original.

The secret sauce — data, not architecture

CLIP's secret was the WIT (WebImageText) dataset:

• 400 million $(image,text)$ pairs scraped from the public internet.
• Constructed using 500k search queries, with up to 20k pairs per query.
• Word count comparable to GPT-2's training corpus.

CLIP's architectural contribution was modest. Its data contribution was unprecedented. Memorise: why does CLIP work? → scale of natural-language supervision, 400M pairs.

Zero-shot classification — the magic trick

Given a new image classification task, no fine-tuning needed:

1. Embed each class name as text: "a photo of a {class}". 2. Encode all text prompts → text embeddings. 3. Encode the input image. 4. Compute cosine similarity image-vs-each-text. 5. Predict the class with the highest similarity.

CLIP zero-shots ImageNet at ~76% top-1 — competitive with a fully-supervised ResNet-50 that was trained on ImageNet.

CLIP's limitations — the "Why?" slide

• Not aware of fine-grained details — colours, exact part attributes.
• Not compositional — "person on horse" vs "horse on person" — CLIP scores them similarly because it matches global representations, not structure.
• Matching global representations is insufficient for spatial relationships.
• Needs hard negatives for compositional reasoning.

Two papers diagnose:

• Winoground (Thrush et al., CVPR 2022) — captions differ only in word order; CLIP performs near chance.
• CLIP Association Bias (Yamada et al., EMNLP 2023, "When are Lemons Purple?") — CLIP sometimes associates "lemons" with "purple" because of textual co-occurrence in training data.

CLIP's impact

• CLIP's visual encoder is one of the most-used vision encoders today.
• LAION-5B — an open-source 5-billion-pair replication of WIT.
• BLIP, InstructBLIP — CLIP-style ideas with added captioning + instruction-tuning.
• Stable Diffusion — uses CLIP's text encoder to condition image generation. Every Stable Diffusion image you see was generated by a model that uses CLIP inside.
• SigLIP — directly succeeds CLIP, swaps softmax for sigmoid.

CLIP is, in many ways, the single most consequential vision paper of 2021 — and the bridge between vision and language.

What you carry into the exam

The four families of SSL. The Gelato Bet. SimCLR's four components and aggressive-augmentation finding. The NT-Xent ≡ softmax-CE connection. MoCo's queue + momentum encoder + the trade-off vs SimCLR. BYOL's predictor + stop-grad + EMA and why it doesn't collapse. SwAV's Sinkhorn-Knopp equipartition. The three motivations for language supervision. DeViSE as historical predecessor. The CLIP pseudocode end-to-end, especially the symmetric loss. WIT's 400M-pair scale. The zero-shot recipe. CLIP's compositional failures and the Winoground / lemons-purple results. The downstream impact list.

Every multimodal model in your course flows downstream from these ideas.