Computer Vision

Computer vision is the science of extracting meaning from images — answering the questions What? Where? Who? When? Why? How? and How many? from a 2D array of pixel intensities. The dirty secret of the field is that pixels are just numbers; everything else (objects, edges, intentions) must be COMPUTED. More than 50% of the human brain is devoted to vision because the problem is computationally vast, not because eyes are complicated.

DIP is the bag of operations that act *on* the pixels — before any 'learning'. Three big paradigms: **spatial** (operate on pixel neighbourhoods directly), **transform** (move to a different basis like Fourier/DCT, operate there, transform back), and **morphological** (set-theoretic operations on binary images via a structuring element). Every modern CNN op (conv, pooling, padding, BN) has a DIP ancestor; understanding the ancestor explains the descendant.

ML in this course is the bag of mathematical tools that turn data into a function. Three task families: **supervised** (X → Y; classification, regression), **unsupervised** (X alone; clustering, density, dimensionality reduction), **reinforcement** (X → action with reward; not the focus here). The key tools — logistic regression, the multilayer perceptron with backprop, PCA, k-means — show up inside every modern CV system as building blocks (a softmax head IS multinomial logistic regression; a CNN backbone IS hierarchical MLP with weight sharing).

A convolution layer is a stack of learned filters slid across an image — a CNN is a hierarchy of these, with each layer composing simpler features (edges → texture → parts → objects). The reason CNNs work in vision: WEIGHT SHARING (one filter, all spatial locations) gives **translation equivariance** (a shifted input produces a shifted output) and slashes parameters by ~1000× vs a dense network. The decade of CNN architecture research (2012–2020) was about how to go deeper, wider, or more efficient without breaking trainability.

Detection answers two questions per object: 'where' (a bounding box) and 'what' (a class). The history of the field is essentially the story of moving the expensive parts of that pipeline — region proposals, per-region computation, ranking — onto the GPU and into a single shared backbone. By 2016 the two-stage R-CNN family had matured (R-CNN → Fast → Faster); YOLO then asked 'why two stages at all?' and reframed detection as a single regression problem in one CNN forward pass.

Detection labels regions; dense prediction labels every pixel. The architectural challenge is restoring spatial resolution lost during the encoder's downsampling — either via transposed convolutions (learnable upsampling), skip connections that re-inject shallow high-resolution features (U-Net), or dilated convolutions that grow the receptive field without losing resolution (DeepLab). Beyond segmentation, the same dense-prediction framework handles depth and surface normals.

A bounding box around a person tells you almost nothing about what they're doing — dancing, fighting, lifting a mug, falling. The answer lives in the body itself, in the geometry of arms, legs, torso, head. Pose estimation predicts joint locations and limbs so downstream tasks (activity recognition, motion capture, gesture interfaces, avatar animation) become tractable. The core architectural shift is from direct keypoint regression (broken for several reasons) to dense prediction via per-joint heatmaps.

A point cloud is a bag of $(x, y, z)$ samples — no grid, no ordering, no fixed count. A 2D CNN needs all three to work. So the field forks into two strategies: either force the data onto a grid and pay for it (VoxNet), or invent operators that are *symmetric* (permutation-invariant) by construction so they see a set, not a sequence (PointNet → PointNet++ → DGCNN → MeshCNN).

Given 100 phone photos of your living room, can a computer reconstruct the 3D scene and render it from a new viewpoint? **NeRF** (2020) said yes by training a tiny MLP $f_\theta(x, y, z, \theta, \phi) \to (\text{RGB}, \sigma)$ and integrating along camera rays — beautiful but slow (hours to train, 1 fps to render). **3D Gaussian Splatting** (2023) kept the *fit-pictures-to-a-scene* objective but threw away the MLP, replacing it with a few million explicit 3D Gaussians that get rasterised in real time. There are no neural network weights and no learning across scenes — every new scene is a fresh per-scene *optimisation*.

Pre-2014 sequence-to-sequence models squeezed an entire source sentence through a single fixed vector — the *bottleneck*. **Attention** (Bahdanau 2015) let the decoder *look back* over the encoder's full history at each step, picking out the positions relevant to the word being generated. Three years later, **Vaswani et al. 2017** asked the radical question: do we need the RNN at all? *Attention Is All You Need* — and every modern vision/language model from ViT to GPT to PaliGemma descends from that one paper.

For 9 years (AlexNet 2012 → 2021), every state-of-the-art vision model was a CNN. The locality + translation-equivariance inductive bias of convolutions was treated as a truth about how vision must work. Then Google's *"An Image is Worth 16×16 Words"* (Dosovitskiy et al., ICLR 2021) showed that a plain Transformer — the same architecture used to translate English to French — matched or beat the best CNNs on ImageNet, *provided you fed it enough data*. The recipe is literally in the title: cut the image into $16 \times 16$ patches, treat each as a token, feed a standard Transformer encoder.

Every method before this needed *labels* — annotated boxes, pixel masks, class names. Then somebody asked: *what if the supervision could come from the data itself?* You hand the network an image with no label, but you create a task that the image can answer with its own structure — *"these two random crops came from the same image"*. Solve enough of these synthetic puzzles and the resulting features generalise. **This is self-supervised learning (SSL)**, and over five years it has become the dominant pretraining recipe in vision. CLIP, DINO, MAE — every modern vision encoder is SSL-pretrained.

The previous unit's contrastive recipe — pull positives, push negatives — works but has problems: it needs many negatives (big batches or memory banks), is sensitive to augmentation choice, and produces good-but-not-great features. In 2021, three papers in quick succession asked: **do we even need negative samples?** One by one they showed you don't. **DINO** (self-distillation), **MAE** (masked reconstruction in pixel space), and **JEPA** (prediction in representation space) form the second great family of SSL alongside contrastive methods.

The 2017 Transformer is still the architecture every state-of-the-art LLM and ViT runs on today. *Broadly speaking* — but "broadly speaking" hides a lot of work. Almost every component has been quietly **replaced** between 2017 and 2025. Not torn down, refined. Same skeleton, modernised organs. The slide pack is named *"ViT-5: Vision Transformers for The Mid-2020s"* — and the framing is exactly that. **Nine** specific upgrades, grouped into four themes: **stability** (Pre-Norm, LayerScale), **normalisation** (RMSNorm, QK-Norm), **efficiency** (Flash Attention, KV-cache, GQA), **positional encoding** (RoPE), and **attention hygiene** (Registers). Knowing what each one does and why is the entire exam.

For most of computer vision history, *image* and *text* lived in separate universes. Vision built models that ate pixels and spit out class labels. NLP built models that ate tokens and spit out tokens. The two never talked. Then **CLIP → BLIP → Flamingo → LLaVA** taught a single network to fuse them. The recipe in 2024 settled into the **three-pillar VLM**: pretrained vision encoder, small connector, pretrained LLM. Show it a photo and ask in English. Hand it an X-ray and ask the diagnosis. Tell it *"pick up the cup"* and have a robot arm move. **PaliGemma → Qwen2-VL → Gemma 4** is three generations of this architecture, each fixing the last's bottleneck.

For thirteen units, you've extracted meaning from a single frozen rectangle of pixels. But the world doesn't hold still. A person *walking* is not the same as a person *standing*. A glass *falling* is not the same as a glass *resting*. **Time is the missing dimension.** A video is a stack of images — technically — but that's like saying a sentence is a stack of letters. Video understanding asks: how do you teach a network to see motion, sequence, and intent? **Two halves to treat as separate exam topics**: *(1)* the **landscape of tasks and datasets**, and *(2)* the **architectural families** that solve them.