Intuition

For most of computer vision history, *image* and *text* lived in separate universes. Vision built models that ate pixels and spit out class labels. NLP built models that ate tokens and spit out tokens. The two never talked. Then CLIP → BLIP → Flamingo → LLaVA taught a single network to fuse them. The recipe in 2024 settled into the three-pillar VLM: pretrained vision encoder, small connector, pretrained LLM. Show it a photo and ask in English. Hand it an X-ray and ask the diagnosis. Tell it *"pick up the cup"* and have a robot arm move. PaliGemma → Qwen2-VL → Gemma 4 is three generations of this architecture, each fixing the last's bottleneck.

Explanation

The Modality Gap — the fundamental problem. Text is discrete: tokens are integers in $\{0,1,\dots ,|V|-1\}$; embedding via lookup $e=W_{\text{emb}}[t]\in {\mathbb{R}}^d$; vocab ~32k for most LLMs. Images are continuous: $x\in {\mathbb{R}}^{H\times W\times C}$ with pixel values in $[0,255]$; you cannot index a lookup table — must run a *learned encoder*. The goal of every VLM: $f_{\text{text}}(w)\approx f_{\text{image}}(x)$ when $w$ describes $x$ — both modalities mapped to the same ${\mathbb{R}}^T$ space with semantic correspondence preserved. Everything else is engineering.

The Three-Pillar Blueprint — every VLM in this unit follows it. *Vision Encoder* → *Connector / Adapter* → *LLM Backbone*. One equation to memorise verbatim: $y=\text{LLM}\left(\text{Concat}\right(\text{Connector}(\text{Encoder}(x_{\text{image}})),\text{Tokenizer}(x_{\text{text}})\left)\right)$. Visual tokens and text tokens are concatenated into one sequence, then fed to a standard autoregressive LLM. *There is no special cross-attention mechanism — vanilla self-attention across the joint sequence does all the cross-modal work.* Exam-gold: if asked *"how does a VLM perform cross-modal reasoning?"*, the answer is self-attention over the concatenated sequence handles it naturally.

PaliGemma — the canonical didactic VLM (Beyer et al., 2024). Three components: *Vision Encoder* SigLIP-So400m (~400M params, frozen in Stage 1); *Connector* a single linear layer, randomly initialised; *LLM* Gemma-2B decoder (pretrained). Total < 3B parameters. Image tokens produced: 256 / 1024 / 4096 for input resolutions 224 / 448 / 896 respectively.

SigLIP vs CLIP — the loss difference. Same setup as CLIP: two encoders $f_I,f_T$, shared ${\mathbb{R}}^d$, batch of $N$ image-caption pairs, matched diagonal should be high-similarity, off-diagonal low. Difference: CLIP uses *softmax* normalised over the full batch — every pair's logit is normalised against the entire row (synchronisation cost; loss landscape depends on batch size). SigLIP uses independent sigmoid binary cross-entropy for every pair. Step 1: pairwise logits $z_{ij}=\tau \cdot f_I(x_i{)}^{\top }{f}_T(t_j)+b$ where $\tau$ is learnable temperature and $b$ a learnable bias. Step 2: $L=-\frac{1}{n^2}{\sum }_i{\sum }_j[y_{ij}\log \sigma (z_{ij})+(1-y_{ij})\log (1-\sigma (z_{ij}))]$ with $y_{ij}=1$ iff $i=j$. SigLIP scales to arbitrary batch size without instability. *CLIP softmax → SigLIP sigmoid* is the key transition.

Inside SigLIP — ViT patch tokenisation. Step 1 — patch the image at $P=14$: $N=(H/P)(W/P)$; for $224\times 224$ input: $N=16^2=256$ patches. Step 2 — embed each patch: $z_i=p_i\cdot E+e_i^{\text{pos}}$ with $E\in {\mathbb{R}}^{P^{2}C\times D}$, $D=1152$ for SigLIP-So400m. Step 3 — Transformer pass: $H=\text{ViT}([z_1,\dots ,z_N])\in {\mathbb{R}}^{N\times D_{\text{enc}}}$. **Memorise: $N=256$, $D_{\text{enc}}=1152$.**

The Connector — a single linear layer. SigLIP outputs 1152-d vectors; Gemma-2B expects 2048-d token embeddings. $v_i=W_{\text{proj}}{H}_i+b_{\text{proj}}$ with $W_{\text{proj}}\in {\mathbb{R}}^{2048\times 1152}$. **Critical exam point: $W_{\text{proj}}$ is the ONLY randomly initialised component of PaliGemma.** Everything else (SigLIP, Gemma) starts from pretrained checkpoints. The connector is *just a dimension change* that learns how to align two pretrained spaces. Result: visual tokens $V=[v_1,\dots ,v_N]\in {\mathbb{R}}^{N\times D_{\text{llm}}}$ — same shape as text token embeddings.

Prefix-LM — the attention masking strategy. Sequence layout: $[{\text{img}}_1\dots {\text{img}}_N|\text{BOS}|t_1\dots t_P(\text{prompt})|\text{SEP}|s_1\dots s_M(\text{answer})]$. Image + prefix = context (e.g. "caption en"): attend *bidirectionally* — every token in this region sees every other. Suffix = the answer to generate: attend *causally* — each suffix token only sees tokens to its left. Mask $M\in \{0,-\infty {\}}^{(N+P+M)\times (N+P+M)}$: $M_{ij}=0$ if $i,j\in \{\text{image}\cup \text{prefix}\}$ (bidirectional); $M_{ij}=0$ if $i\in \text{suffix}$ and $j\le i$ (causal); $-\infty$ otherwise. Loss is computed ONLY on suffix tokens. Image and prompt are free context — inputs the model doesn't have to predict.

PaliGemma's three training stages. *Stage 1 — unimodal pretraining:* SigLIP pretrained on image-text pairs (contrastive); Gemma pretrained on text. No joint training yet. *Stage 2 — multimodal pretraining:* freeze SigLIP, train $W_{\text{proj}}$ + LLM on a large image-text corpus (WebLI, CC12M, …). Images at $224^2$ → 256 tokens. *Stage 3 — task transfer:* fine-tune on specific tasks (VQA, captioning, detection, segmentation). Unfreeze all components, use higher resolution.

"Everything is text" — PaliGemma's task design. Cleverest design choice: every task output is text, even spatial outputs. *Captioning:* prompt "caption en" → natural sentence. *VQA:* prompt "How many people are visible?" → "3". *Object detection:* prompt "detect person" → "$⟨\text{loc0210}⟩⟨\text{loc0142}⟩⟨\text{loc0867}⟩⟨\text{loc0821}⟩$ person" — four $⟨\text{locXXXX}⟩$ tokens encode $[y_{\text{min}},x_{\text{min}},y_{\text{max}},x_{\text{max}}]$ as normalised coordinates. *Referring segmentation:* bbox tokens + $⟨\text{seg042}⟩⟨\text{seg018}⟩\dots$ codewords from a VQ-VAE codebook. **Vocab extended with 1024 location tokens $⟨\text{loc0000}⟩\dots ⟨\text{loc1023}⟩$ and 128 segmentation codewords.** Exam-gold: *"how does PaliGemma do detection if it only outputs text?"* → *special location tokens in the extended vocabulary.*

Qwen2-VL — fixing two PaliGemma limitations. *Problem 1 — Resolution bottleneck.* SigLIP processes at fixed size (224, 448, 896). A 3mm nodule on a $2048^2$ chest X-ray becomes sub-pixel after resizing. Information-theoretic bound: once downsampled, the detail is irrecoverable. *Problem 2 — Aspect ratio distortion.* A 16:9 photo forced into a 1:1 grid distorts spatial relationships; receipt items appear ~40% shorter. Qwen2-VL fixes both with dynamic resolution + M-RoPE.

Naïve Dynamic Resolution — tile at native aspect ratio. Just stop resizing. Tile the image with patches at its actual size: $N_h=H_{\text{orig}}/P$, $N_w=W_{\text{orig}}/P$, $N=N_h\cdot N_w$ — varies per image. $448\times 224$ with $P=14$: $N=32\cdot 16=512$ tokens. $896\times 896$ with $P=14$: $N=64\cdot 64=4096$ tokens. *Practical:* batch collation requires padding to longest length (or dynamic packing — multiple small images share one sequence window). Inference cost scales with $N$ → user controls the resolution-speed trade-off. Token budget control: $N_{\text{min}}\le N\le N_{\text{max}}$ (clamp; resize minimally to satisfy budget).

1D RoPE breaks for images. Raster index $m=\text{row}\cdot W+\text{col}$. Two spatially adjacent patches at $(2,W-1)$ and $(3,0)$ have $m$ differing by 1 — they *look adjacent* — while $(2,0)$ and $(3,0)$ differ by $W$ — they *look far apart* even though physically adjacent. 1D ordering is incompatible with 2D geometry.

2D-RoPE — encoding spatial position. Split the head dimension $d$ into two halves; apply RoPE independently to each — one for rows, one for columns. Patch at $(r,c)$: rotate first half $[0:d/2]$ by $R_{m_h=r}$ (row position); rotate second half $[d/2:d]$ by $R_{m_w=c}$ (column position). **Attention score now encodes 2D relative displacement $(\Delta r,\Delta c)$.** Invariant to image size — generalises to any resolution and aspect ratio.

M-RoPE — Multimodal RoPE (time × height × width). For video, extend to a *third* dimension. Split head dim $d$ into thirds. Token at $(t,r,c)$: rotate $[0:d/3]$ by $R_{m_t}$ (temporal); $[d/3:2d/3]$ by $R_{m_r}$ (row); $[2d/3:d]$ by $R_{m_w}$ (col). Edge cases: *static images* — $m_t=0$ for all patches. *Text tokens* — same $m$ for all three dimensions: $m_t=m_r=m_c=\text{token\_index}$. Why this is the cleverest part: attention between two video patches now depends on $(t_1-t_2,r_1-r_2,c_1-c_2)$ — spatio-temporal displacement. The model can reason *"how did the object at $(r,c)$ move from frame $t_1$ to $t_2$?"*

Qwen2-VL vs PaliGemma — the architectural delta table. *Vision encoder:* SigLIP-So400m (400M) vs ViT 600M + 2D-RoPE. *Connector:* linear $W\in {\mathbb{R}}^{2048\times 1152}$ vs 2-layer MLP ($D_{\text{enc}}\to 4D_{\text{enc}}\to D_{\text{llm}}$). *PE:* 1D-RoPE vs M-RoPE ($T\times H\times W$). *Token count:* fixed 256/1024/4096 vs dynamic $N_h\times N_w$ (up to 32k for video). *LLM:* Gemma 2B vs Qwen2 7B/72B. Qwen2-VL-72B matched GPT-4o on most multimodal benchmarks (Oct 2024).

Gemma 4 — close the connector seam. Even Qwen2-VL has a residual problem: vision and language are *still* trained in separate spaces before being stitched. The connector remains an information bottleneck between two latent geometries optimised for *different* objectives — SigLIP for image-text cosine similarity (contrastive); Gemma for next-text-token prediction (LM). A linear connector can only rotate and scale; a 2-layer MLP is more expressive but still a narrow bridge between two independent latent spaces. Gemma 4's answer: native multimodal training from scratch. Vision and language share Transformer blocks from early layers. Visual patches and text tokens are processed by the *same* weight matrices. No dedicated projection layer — modalities meet inside the Transformer. The lecture's framing: *stitched → woven*. Architecture variants: Gemma 4B / 12B / 27B; 128k context; 32-frame video clips inline.

The three-generation comparison table — likely exam target. *Vision Encoder:* SigLIP-So400m (400M) → ViT + 2D-RoPE (600M) → native / integrated. *Resolution:* fixed 224/448/896 → dynamic (native AR) → dynamic, multi-scale. *Connector:* linear $W$ → 2-layer MLP → deep fusion (none). *Positional Encoding:* 1D-RoPE (text only) → M-RoPE $(T,H,W)$ → unified spatial RoPE. *Position math:* $m\in \mathbb{Z}$ → $(m_h,m_w)\in {\mathbb{Z}}^2$ → $(m_t,m_h,m_w)\in {\mathbb{Z}}^3$. *Token Count:* fixed → dynamic $N_h\times N_w$ → dynamic multi-res. *Training:* frozen enc + tuned proj → joint tuning → native joint training.

From VLMs to VLAs — when models must act. A VLM can describe a coffee cup in detail; it cannot pick it up. Closing this gap is the point of Vision-Language-Action (VLA) models — robots driven by LLMs. The key insight: actions are just another type of token. Extended vocab $V^{\prime }=V\cup \{\Delta x_{{\text{bin}}_0},\dots ,\Delta {\text{grip}}_{{\text{bin}}_{255}}\}$. Each continuous action dimension discretised into 256 bins. A 7-DOF arm $(\Delta x,\Delta y,\Delta z,\Delta \text{roll},\Delta \text{pitch},\Delta \text{yaw},\Delta \text{grip})$ → $7\times 256=1792$ new tokens. OpenVLA reuses the 256 least-frequent text tokens for action bins to avoid expanding the embedding table.

Action tokenisation — the math. Discretise each dim with range $[a_{\text{min}},a_{\text{max}}]$ and $B=256$ bins: $\text{bin}(a_d)=\lfloor (a_d-a_{\text{min}})/(a_{\text{max}}-a_{\text{min}})\cdot B\rfloor$. De-tokenise with midpoint: ${\hat{a}}_d=a_{\text{min}}+(b+0.5)/B\cdot (a_{\text{max}}-a_{\text{min}})$. The $+0.5$ centres within the bin. **Quantisation error bounded by $(a_{\text{max}}-a_{\text{min}})/(2B)$.**

OpenVLA architecture (Kim et al., 2024). Two vision encoders concatenated: DinoV2 (strong spatial features, good for manipulation) + SigLIP (strong semantic features, good for instruction following). MLP projector combines them. LLaMA-2 7B backbone. Action de-tokeniser at the output: the last 7 logits are over 256 action bins per dimension; argmax → bin → continuous action. Trained on 970k robot trajectories from Open-X-Embodiment; ~7B params; fine-tunable on 1–2 hours of single-task demonstrations.

VLA training objective. Initialise from a pretrained VLM (inherits language + vision understanding); fine-tune on robot trajectories with a joint loss: $L=L_{\text{LM}}+\lambda \cdot L_{\text{action}}$ where $L_{\text{LM}}=-{\sum }_t\log P(y_t\mid x_{\text{image}},\text{instr},y_{<t})$ (text CE — keeps language ability) and $L_{\text{action}}=-{\sum }_t{\sum }_d\log P(a_{d,t}\mid x_{\text{image}},\text{instr},a_{<t})$ (CE on each discretised action dim).

The full arc — perceive → align → reason → generate → act. *Perceive:* ViT patches → $N$ feature vectors. *Align:* connector → same space as text tokens. *Reason:* LLM attends over $[V\Vert \text{text}]$ with M-RoPE. *Generate:* autoregressive decoder outputs text OR action tokens. *Act:* action de-tokeniser → continuous deltas. PaliGemma / Qwen2-VL / Gemma 4 cover steps 1–4; OpenVLA / RT-2 extend to step 5.

Definitions

• Modality gap — Text is discrete (vocab → lookup); images are continuous (pixels → encoder). A VLM aligns them into a shared latent space.
• Three-pillar blueprint — Vision Encoder → Connector → LLM Backbone. Visual + text tokens concatenated, vanilla self-attention handles cross-modal reasoning.
• Connector / adapter — Small (linear or MLP) projection from vision-encoder output dim to LLM token dim. In stitched VLMs, the only randomly initialised component.
• SigLIP — Sigmoid CLIP. Pairwise binary cross-entropy with logits $z_{ij}=\tau f_I(x_i{)}^{\top }{f}_T(t_j)+b$. Scales without batch-wide softmax sync.
• Prefix-LM mask — Bidirectional attention over image + prompt; causal over answer. Loss computed only on answer tokens.
• Location tokens — 1024 extended-vocab tokens $⟨\text{loc0000}⟩\dots ⟨\text{loc1023}⟩$ encoding normalised bounding-box coordinates. Detection output is pure text.
• Segmentation codewords — 128 extended-vocab tokens $⟨\text{seg000}⟩\dots ⟨\text{seg127}⟩$ from a learned VQ-VAE codebook. Segmentation mask = bbox + codeword sequence decoded to pixels.
• Dynamic resolution (Qwen2-VL) — Process at native aspect ratio + resolution; tile-count clamped by $N_{\text{min}}\le N\le N_{\text{max}}$. Fixes resolution bottleneck + aspect distortion.
• 2D-RoPE — Split head dim in halves; rotate first half by row, second half by column. Attention depends on 2D relative displacement.
• M-RoPE — Multimodal RoPE (Qwen2-VL): head dim split in thirds for $(t,r,c)$ rotations. Static images: $m_t=0$. Text tokens: all three = token index.
• Native multimodal (Gemma 4) — Vision and language share Transformer blocks from early layers; no connector. Modalities meet inside the Transformer with shared weights.
• VLA (Vision-Language-Action) — VLM + action de-tokeniser. Continuous actions discretised into 256 bins per dimension; treated as just another token type.
• OpenVLA — DinoV2 + SigLIP image encoders + MLP connector + LLaMA-2 7B + action de-tokeniser. Trained on 970k robot trajectories. ~7B params.

Formulas

• $$y=\text{LLM}\left(\text{Concat}\right(\text{Connector}(\text{Encoder}(x_{\text{image}})),\text{Tokenizer}(x_{\text{text}})\left)\right)$$
• $$\text{SigLIP logits:}{z}_{ij}=\tau \cdot f_I(x_i{)}^{\top }{f}_T(t_j)+b$$
• $$\text{SigLIP loss:}L=-\frac{1}{n^2}\sum _{i,j}[y_{ij}\log \sigma (z_{ij})+(1-y_{ij})\log (1-\sigma (z_{ij}))]$$
• $$\text{ViT patch:}N=(H/P)(W/P);z_i=p_i\cdot E+e_i^{\text{pos}},E\in {\mathbb{R}}^{P^{2}C\times D}$$
• $$\text{Linear connector:}{v}_i=W_{\text{proj}}{H}_i+b,W_{\text{proj}}\in {\mathbb{R}}^{D_{\text{llm}}\times D_{\text{enc}}}$$
• $$\text{Prefix-LM mask:}{M}_{ij}=\{\begin{array}{ll}0& i,j\in \{\text{img}\cup \text{prefix}\}\\ 0& i\in \text{suffix},j\le i\\ -\infty & \text{otherwise}\end{array}$$
• $$\text{Dynamic token count:}{N}_{\text{min}}\le N=N_h\cdot N_w\le N_{\text{max}}$$
• $$\text{2D-RoPE:}\text{rotate}[0:d/2]\text{by}{R}_r;\text{rotate}[d/2:d]\text{by}{R}_c$$
• $$\text{M-RoPE position triple:}(t,r,c).\text{Static image:}t=0.\text{Text token:}t=r=c=\text{idx}$$
• $$\text{VLA action bin:}\text{bin}(a_d)=\lfloor \frac{a_d-a_{\text{min}}}{a_{\text{max}}-a_{\text{min}}}\cdot B\rfloor ,B=256$$
• $$\text{VLA de-bin:}{\hat{a}}_d=a_{\text{min}}+\frac{b+0.5}{B}(a_{\text{max}}-a_{\text{min}})$$
• $$\text{VLA loss:}L=L_{\text{LM}}+\lambda \cdot L_{\text{action}}=\text{CE}(\text{text})+\lambda \cdot \text{CE}(\text{action})$$

Derivations

Why CLIP softmax doesn't scale and SigLIP sigmoid does. CLIP loss for one image $i$: $-\log \frac{\exp (z_{ii})}{{\sum }_j\exp (z_{ij})}$ — denominator sums over *all $n$ candidates in the batch*, so the gradient for image $i$ depends on every other text $t_j$. At distributed scale, this requires an all-to-all softmax sync across GPUs every step, and the optimisation landscape changes with batch size. SigLIP replaces this with ${\sum }_{i,j}\text{BCE}(\sigma (z_{ij}),y_{ij})$ — each pair is its own independent binary problem, the loss factorises, no all-to-all sync, and batch size only affects how many *examples* the gradient sees, not the loss *shape*. SigLIP scales to arbitrarily large batches without instability.

Why Prefix-LM beats pure causal for VLMs. With *causal* masking on the entire sequence, image patches can only attend to previously processed patches — and the order is artificial (raster scan). The encoder cannot fully digest the image before the LM starts generating. With *Prefix-LM*, the image + prompt region is bidirectional (encoder-like: every patch sees every patch + every prompt token); only the answer is causal. The model gets *encoder-quality understanding of the input* AND autoregressive generation — best of both. Loss is on the answer only, so the bidirectional region doesn't violate the LM objective.

Why 1D RoPE breaks images, with concrete math. Image of size $H\times W$ patches. Raster index: $m(r,c)=r\cdot W+c$. Two physically adjacent patches *(2, W-1)* and *(3, 0)*: $|m_1-m_2|=1$, but they're in different rows. Two physically distant patches *(2, 0)* and *(3, 0)* (same column, adjacent row): $|m_1-m_2|=W$. The 1D ordering says 1 patch (across a row wrap-around) is closer than 1 patch (across a row boundary), which is geometrically nonsense. 2D-RoPE separates row and column dimensions so adjacency in either direction is encoded correctly.

Why the connector is the bottleneck in stitched VLMs. SigLIP's latent geometry was shaped by a contrastive objective: nearby vectors mean *visually similar*. Gemma's latent geometry was shaped by a language-modelling objective: nearby vectors mean *similar next-token distribution*. These geometries are not isometric. A linear map can only rotate and scale — at best, it aligns axes; it cannot remap *non-linear* semantic relationships. A 2-layer MLP is more expressive but still a narrow bridge. Gemma 4's solution: don't reconcile separate geometries — train them jointly from scratch so they share a geometry.

Quantisation error in VLA action discretisation. Bin width $w=(a_{\text{max}}-a_{\text{min}})/B$. Midpoint de-bin: $\hat{a}=a_{\text{min}}+(b+0.5)/B\cdot w\cdot B=a_{\text{min}}+(b+0.5)w$. Max error: half a bin width, $w/2=(a_{\text{max}}-a_{\text{min}})/(2B)$. For $B=256$ and a 1-meter arm range, max error is ~2 mm — fine for most manipulation.

Examples

• **PaliGemma forward pass on a $224^2$ image.** Image → SigLIP-So400m → $16\times 16=256$ patch features (each 1152-d) → linear connector $W\in {\mathbb{R}}^{2048\times 1152}$ → 256 patch tokens (2048-d) → concat with text → Gemma decoder under Prefix-LM mask → causal next-token loss on suffix only.
• SigLIP loss on a batch of 4. $z_{ii}$ on the diagonal should be large; $z_{ij}$ off-diagonal should be small. The loss treats each of the 16 pairs independently as a binary classification (is this pair matched?). No batch-wide softmax sync.
• Detection through text. Input: $224^2$ image of a dog + prompt "detect dog". Output: "$⟨\text{loc0142}⟩⟨\text{loc0210}⟩⟨\text{loc0821}⟩⟨\text{loc0867}⟩$ dog". The four location tokens decode to $[y_{\text{min}},x_{\text{min}},y_{\text{max}},x_{\text{max}}]=[0.139,0.205,0.802,0.847]$ in normalised coords. The model never outputs floats — only tokens.
• **M-RoPE for a video frame at $(t=5,r=2,c=3)$.** Rotation matrices applied to three thirds of the head dim by angles $5\theta ,2\theta ,3\theta$ respectively. Attention between this token and another at $(t=8,r=2,c=3)$ depends only on $(\Delta t,\Delta r,\Delta c)=(3,0,0)$ — purely temporal displacement.
• **Qwen2-VL with $N_{\text{min}}=256,N_{\text{max}}=1280$.** A $512\times 768$ image at $P=14$ → $N\approx 36\times 54=1944$ patches → clamped down. A $2048\times 3072$ image → still clamped to $N_{\text{max}}=1280$. User controls the cost-quality trade-off explicitly.
• OpenVLA inference. Image of mug + instruction "pick up the red mug". DinoV2 + SigLIP encode the image; MLP projects; LLaMA-2 generates: ${\text{tok}}_1,{\text{tok}}_2,\dots$ (action bins for $\Delta x,\Delta y,\Delta z,\Delta \text{roll},\dots ,\Delta \text{grip}$). De-tokeniser converts to continuous action. Repeat at 10 Hz.
• VLA bin error. 7-DOF arm; $a_x\in [-0.5,0.5]$ m; $B=256$. Bin width: $1.0/256\approx 3.9$ mm. Max error: ~2 mm. Fine for pick-and-place; not for surgical precision.

Diagrams

• Three-pillar VLM (PaliGemma). SigLIP (frozen, 400M) → linear connector (random init, the only trained-from-scratch piece) → Gemma decoder (pretrained, 2B). Annotate frozen vs trained vs random-init.
• Prefix-LM attention mask. A $(N+P+M)\times (N+P+M)$ matrix. Upper-left $(N+P)\times (N+P)$ block: all zeros (bidirectional). Suffix region: lower-triangular zeros (causal). Cross blocks: image+prefix can see suffix? No (zeros only where allowed).
• SigLIP vs CLIP loss. Side-by-side. CLIP: $n\times n$ matrix → row softmax + col softmax → average. SigLIP: $n\times n$ matrix → element-wise sigmoid + BCE per cell → average.
• ViT patch tokenisation. $224\times 224$ image → $16\times 16$ grid of $14\times 14$ patches → flatten + linear → 256 tokens of dim 1152.
• Detection-through-text. Image + prompt "detect cat" → output sequence "$⟨\text{locA}⟩⟨\text{locB}⟩⟨\text{locC}⟩⟨\text{locD}⟩$ cat" → decode locs to bbox.
• M-RoPE head-dim split. A 64-dim head split into three groups of (e.g.) 22 dims; each group rotated by its own $m_t,m_r,m_c$.
• Three-generation evolution. Timeline: PaliGemma (linear) → Qwen2-VL (MLP + M-RoPE + dynamic) → Gemma 4 (woven, no connector).
• VLA action de-tokeniser. Last 7 logit outputs → argmax per dim → 7 bin indices → midpoint formula → continuous 7-DOF action.
• The five-stage arc. Perceive (ViT) → Align (connector) → Reason (LLM self-attention) → Generate (decoder) → Act (de-tokeniser).

Edge cases

• **PaliGemma at $224^2$:** fine detail (3mm nodule on $2048^2$ X-ray) is irrecoverably lost — downsampling is one-way.
• Aspect-ratio distortion: resizing a 16:9 photo to 1:1 stretches horizontally; receipt items appear ~40% shorter.
• Connector bottleneck: a linear map can only rotate/scale; non-linear semantic relationships across the modality gap may not survive.
• Qwen2-VL batch padding: mixed-resolution batches require padding to longest length or dynamic packing; naïve implementations waste compute.
• M-RoPE edge cases: static images use $m_t=0$; text tokens use $m_t=m_r=m_c=\text{token\_idx}$ — this allows text and image positions to share the rotation but with degenerate spatial structure for text.
• Action quantisation error is half-bin-width at midpoint de-bin. For very fine manipulation (sub-mm), 256 bins per dim may be insufficient.
• **SigLIP bias term $b$** is necessary because the sigmoid output range is $[0,1]$ — without $b$, all $z_{ij}$ near zero produce $\sigma (0)=0.5$ uniformly, hard to discriminate.

Common mistakes

• Saying *"CLIP and SigLIP have the same loss with different normalisation"* — SigLIP is pairwise sigmoid BCE, fundamentally independent across pairs.
• Forgetting that loss is computed ONLY on suffix tokens in PaliGemma — image+prompt tokens don't contribute to the loss.
• Treating M-RoPE as "RoPE with three encodings stacked" — it splits the head dim into thirds and rotates each by a different position.
• Claiming Qwen2-VL processes $224^2$ fixed — it processes native resolution clamped by $N_{\text{min}}/N_{\text{max}}$.
• Saying VLMs have a *"cross-attention mechanism"* — they don't. Vanilla self-attention over the concatenated sequence does all the cross-modal work.
• Stating PaliGemma outputs *bounding box coordinates as floats* — **it outputs $⟨\text{locXXXX}⟩$ tokens**; coordinates are decoded from token indices.
• Confusing prefix (image + prompt) with suffix (answer) in Prefix-LM; only the suffix is autoregressive and only it contributes to the loss.
• Stating Gemma 4 *"uses an MLP connector"* — Gemma 4 is native multimodal, no connector at all.

Shortcuts

• Three pillars to recite: Vision Encoder + Connector + LLM.
• Connector = only random-init component in stitched VLMs.
• Prefix-LM: bidirectional on prompt, causal on answer, loss on answer only.
• SigLIP: pairwise sigmoid BCE; CLIP: softmax row+col CE.
• **M-RoPE = $(t,r,c)$ triple**, head dim split into thirds. Text: all three = token idx.
• Detection through text: $⟨\text{loc0000}⟩\dots ⟨\text{loc1023}⟩$ encode normalised $[y_{\text{min}},x_{\text{min}},y_{\text{max}},x_{\text{max}}]$.
• Three generations: PaliGemma (linear, 1D-RoPE, fixed) → Qwen2-VL (MLP, M-RoPE, dynamic) → Gemma 4 (woven, native).
• VLA = VLM + action de-tokeniser; 7-DOF × 256 bins.

Proofs / Algorithms

SigLIP loss factorises across pairs. $L=-\frac{1}{n^2}{\sum }_{i,j}\text{BCE}(\sigma (z_{ij}),y_{ij})$ — the inner sum over $j$ is independent of $i$ at the loss-shape level (each term is its own binary CE). In contrast, CLIP's softmax loss $-\log \frac{\exp (z_{ii})}{{\sum }_j\exp (z_{ij})}$ has every term depending on the partition ${\sum }_j$ — adding negatives changes the loss landscape. Hence SigLIP scales to arbitrary $n$ without batch-size-dependent dynamics.

Prefix-LM preserves the autoregressive property of the suffix. For $i\in \text{suffix}$, the mask allows attention only to $j\le i$. Cross-entropy is computed only at suffix positions $i$, where the prediction ${\hat{y}}_i$ depends only on tokens at positions $\le i$. So the conditional probability factorisation $p(\text{suffix}\mid \text{prefix})={\prod }_{i}p(y_i\mid y_{<i},\text{prefix})$ holds, making it a valid LM despite the bidirectional prefix region.

M-RoPE preserves spatio-temporal relative position. With head dim split into three groups, the attention score is ${\tilde{q}}^{\top }\tilde{k}={\sum }_{g\in \{t,r,c\}}{q}_g^{\top }{R}_{m_g}^{\top }{R}_{n_g}{k}_g={\sum }_g{q}_g^{\top }{R}_{n_g-m_g}{k}_g$. Each group contributes a function of $(n_g-m_g)$ for $g\in \{t,r,c\}$ — hence the score depends only on the relative triple $(\Delta t,\Delta r,\Delta c)$, not absolute positions. Generalises across resolutions, aspect ratios, and video lengths.