Intuition

ML in this course is the bag of mathematical tools that turn data into a function. Three task families: supervised (X → Y; classification, regression), unsupervised (X alone; clustering, density, dimensionality reduction), reinforcement (X → action with reward; not the focus here). The key tools — logistic regression, the multilayer perceptron with backprop, PCA, k-means — show up inside every modern CV system as building blocks (a softmax head IS multinomial logistic regression; a CNN backbone IS hierarchical MLP with weight sharing).

Explanation

Train / Val / Test. Train: fit weights. Val: tune hyperparameters (model size, learning rate, regularisation strength). Test: report final performance, ONCE. Never tune on test (otherwise test becomes another val and you have no unbiased estimate of generalisation). Curse of dimensionality: in high $d$, distances concentrate, volumes blow up, and naive methods (kNN, KDE) need exponential data to fill the space.

Logistic regression — the canonical classifier. Binary: $\sigma (w^{\top }x+b)\in (0,1)$ where $\sigma (z)=1/(1+e^{-z})$. Loss = binary cross-entropy $L=-y\log \hat{y}-(1-y)\log (1-\hat{y})$. *Not MSE* — MSE is non-convex under sigmoid and gradients vanish. SGD update: $w\leftarrow w-\eta (\hat{y}-y)x$ — note the clean form, identical to linear regression's gradient (this is not coincidence: it's the canonical GLM result). Decision boundary is LINEAR in $x$: $w^{\top }x+b=0$. Multinomial (softmax) for $K$ classes: $P(y=k|x)=e^{w_k^{\top }x}/{\sum }_j{e}^{w_j^{\top }x}$ with cross-entropy loss $-{\sum }_k{y}_k\log {\hat{y}}_k$.

Neural networks. One neuron: $y=\varphi (w^{\top }x+b)$. Stack into layers (MLP). Activations: *sigmoid* $(0,1)$ — saturates both ends, vanishing gradient. *tanh* $(-1,1)$ — zero-centred but still saturates. ReLU $\max (0,x)$ — fast, gradient = 1 in positive half, sparse activations, 'dying ReLU' risk. *Leaky ReLU* and *GELU* fix the death problem.

Backpropagation. Chain rule for gradients through a composition. For each parameter, the gradient is the product of local Jacobians along the path from output back to that parameter. Implementations: auto-diff (PyTorch / TF) build a computation graph at forward time, then traverse it backwards.

Why not zero-initialise weights? SYMMETRY. If $W=0$, every neuron in a layer receives the same input and produces the same output, so backprop drives them with identical gradients → they remain identical forever. The whole layer collapses to one neuron's worth of expressivity. Fix: random init.

Xavier (Glorot) init: $w\sim \mathcal{N}(0,1/n_{\text{in}})$ — for sigmoid/tanh. Kaiming (He) init: $w\sim \mathcal{N}(0,2/n_{\text{in}})$ — for ReLU; the factor of 2 compensates for the half of activations zeroed out.

Speed-up tricks (memorise the six). (1) Batch Normalisation — normalise activations to zero-mean unit-variance per mini-batch, then learnable $\gamma ,\beta$ scale-shift. Faster training, higher LR, less init-sensitive, slight regularisation. (2) Higher LR (BN enables this). (3) Dropout — randomly zero $p$ fraction of activations during training (typically $p=0.5$ in FC, $0.1$–$0.3$ in CNNs). Acts as ensemble + regulariser; at inference, scale by $(1-p)$ or use inverted dropout. (4) Shuffle data between epochs. (5) Less L2 weight decay when BN is present. (6) LR decay / scheduling — start higher, decay over epochs.

Cross-entropy vs MSE for classification. CE is preferred. (a) CE matches the maximum-likelihood interpretation under Bernoulli/categorical. (b) The sigmoid+CE gradient simplifies to $(\hat{y}-y)x$ — no vanishing 'derivative of sigmoid' factor. With MSE+sigmoid, the gradient has an extra ${\sigma }^{\prime }(\cdot )=\sigma (1-\sigma )$ that vanishes at saturation → slow learning.

Ensembles. Bagging = bootstrap aggregating. Sample with replacement, train independent models, average predictions. Reduces VARIANCE. Random Forest is the canonical example. Boosting = sequential, each model focuses on the previous model's errors via reweighting. Reduces BIAS. AdaBoost is the textbook case; Viola-Jones face detection (2001) used AdaBoost over Haar features for real-time detection. Stacking = train a meta-model on the base models' predictions.

Density estimation. *Histogram* — bin-dependent, not smooth, fails in 2D+. *KDE* — Gaussian kernel per data point, sum and normalise. *Gaussian Mixture Model (GMM)*: $p(x)={\sum }_{k=1}^K{\pi }_k\mathcal{N}(x\mid {\mu }_k,{\Sigma }_k)$ with $\sum {\pi }_k=1$. Trained via EM.

EM for GMM. *E-step*: compute soft responsibilities ${\gamma }_{ik}={\pi }_k\mathcal{N}(x_i\mid {\mu }_k,{\Sigma }_k)/{\sum }_j{\pi }_j\mathcal{N}(x_i\mid {\mu }_j,{\Sigma }_j)$ — posterior that point $i$ came from component $k$. *M-step*: update each component by weighted MLE: ${\mu }_k={\sum }_i{\gamma }_{ik}{x}_i/{\sum }_i{\gamma }_{ik}$, ${\Sigma }_k={\sum }_i{\gamma }_{ik}(x_i-{\mu }_k)(x_i-{\mu }_k{)}^{\top }/{\sum }_i{\gamma }_{ik}$, ${\pi }_k=(1/N){\sum }_i{\gamma }_{ik}$. Iterate. Monotonically improves log-likelihood, converges to a local max.

RNNs. Recurrence $h_t=f_{\theta }(h_{t-1},x_t)$; same $\theta$ shared across time. BPTT unrolls through time and backpropagates. Vanishing / exploding gradients: $\partial L/\partial h_0\sim \prod W$; if $\Vert W\Vert <1$ vanishes, if $>1$ explodes. Fix vanishing with LSTM/GRU (additive cell-state path → constant error carousel). Fix exploding with gradient clipping.

LSTM. Three gates (input, forget, output) + cell state $c_t$. $f_t=\sigma (W_f[h_{t-1},x_t])$, $i_t=\sigma (W_i[\cdot ])$, ${\tilde{c}}_t=\tanh (W_c[\cdot ])$, $c_t=f_t\odot c_{t-1}+i_t\odot {\tilde{c}}_t$, $o_t=\sigma (W_o[\cdot ])$, $h_t=o_t\odot \tanh (c_t)$. The additive $f_t\odot c_{t-1}$ path is what gives the constant error carousel.

GRU. Simpler: update gate $z_t$ (merges forget + input) and reset gate $r_t$. No separate cell state. Often as good as LSTM with fewer parameters.

RNN I/O shapes. 1-to-1 (image classification), 1-to-many (image captioning), many-to-1 (sentiment / video classification), many-to-many synced (per-frame labels), many-to-many shifted (translation / video captioning).

Metrics. Confusion matrix → TP, FP, TN, FN. Precision = $TP/(TP+FP)$ — of what I retrieved, how much was right. Recall (TPR) = $TP/(TP+FN)$ — of what was right, how much did I retrieve. F1 = $2PR/(P+R)$ — harmonic mean. Specificity (TNR) = $TN/(TN+FP)$. FPR = $FP/(FP+TN)=1-\text{TNR}$.

When precision vs recall? Monitor PRECISION when FALSE POSITIVES are costly (spam filter: flagging real mail). Monitor RECALL when FALSE NEGATIVES are costly (cancer screening: missing a real tumour).

AP / mAP. AP = area under the PR curve for one class. mAP = mean over classes (or over queries in retrieval). Compute by sorting by score, walking down the list, marking TP/FP at the IoU threshold, building cumulative P and R, and integrating.

ROC vs PR. ROC plots TPR vs FPR. PR plots Precision vs Recall. For BALANCED data, ROC is informative. For IMBALANCED data with rare positives, ROC looks deceptively good (many TNs in the denominator inflate it); PR is preferred because the precision axis directly tracks rare-positive performance.

kNN. Lazy learner: no training, just store. Predict by majority vote among $k$ nearest training points. $k$ odd to avoid ties. **Effect of $k$**: small $k$ → high variance, low bias (overfits, noisy). Large $k$ → low variance, high bias (oversmooths). Distance: Euclidean (L2) — default; Manhattan (L1) — robust; Cosine — directional similarity; Mahalanobis — covariance-aware; Hamming — discrete strings.

Linear regression. $y={\beta }_0+{\beta }_{1}x+ϵ$. Least squares minimises SSE. Closed-form via normal equations: $\hat{\beta }=(X^{\top }X{)}^{-1}{X}^{\top }y$. Fails when $X^{\top }X$ is singular (collinear features) or when $X$ is too large to invert — fall back to gradient descent. Linear in COEFFICIENTS, not variables — that's why polynomial regression $y={\beta }_0+{\beta }_{1}x+{\beta }_2{x}^2$ is still 'linear' (linear in $\beta$).

Regularisation. L1 (Lasso) $\lambda \sum |w|$ — drives weights to exactly zero, performs feature selection. L2 (Ridge) $\lambda \sum w^2$ — shrinks weights smoothly, no exact zeros. Elastic Net combines both.

PCA goal. Find orthogonal directions of maximum variance. Steps: (1) centre data $\tilde{X}=X-\bar{X}$. (2) Covariance $\Sigma ={\tilde{X}}^{\top }\tilde{X}/(n-1)$. (3) Eigen-decompose $\Sigma =V\Lambda V^{\top }$. (4) Keep top-$k$ eigenvectors → projection $Z=\tilde{X}{V}_k$. ${\lambda }_i$ = variance along $v_i$. Reconstruct via $\hat{X}\approx Z{V}_k^{\top }$.

SVD. $X=U\Sigma V^{\top }$. The right singular vectors $V$ = eigenvectors of $X^{\top }X$ = principal components. The singular values ${\sigma }_i=\sqrt{{\lambda }_i}$. Numerically stable (no need to form $X^{\top }X$ which squares the condition number). Low-rank approximation: keep top-$r$ singular values → best rank-$r$ approximation in Frobenius norm (Eckart–Young).

LoRA = low-rank update. $\Delta W=AB$ where $A\in {\mathbb{R}}^{d\times r}$, $B\in {\mathbb{R}}^{r\times d}$, $r\ll d$. Train only $A,B$ — saves memory / compute when fine-tuning huge models. Spiritually the same trick as SVD/PCA.

Eigenfaces. Apply PCA to face images. Top eigenvectors look like ghostly average faces. Represent a new face as a linear combination of top eigenfaces; classify by nearest-neighbour in this low-dim space.

k-means. (a) Init $K$ centres. (b) Assign each point to nearest centre. (c) Recompute centres as mean of assigned points. (d) Repeat until convergence. Issues (memorise four): HARD assignments, favours SPHERICAL equal-size clusters, sensitive to INIT (use k-means++: spread initial centres by sampling proportionally to squared distance from existing centres), sensitive to OUTLIERS (use k-medoids instead).

GMM clustering = soft k-means with shapes. Responsibilities ${\gamma }_{ik}$ provide fractional membership; covariances ${\Sigma }_k$ allow elongated/rotated clusters.

Hierarchical clustering. Agglomerative (bottom-up): start with each point as a cluster, repeatedly merge closest pair. Divisive (top-down): start with one cluster, repeatedly split. Output: a dendrogram — cut at different heights to get different numbers of clusters. Linkage criteria: single (closest pair), complete (farthest pair), average, Ward (minimum variance).

Definitions

• Sigmoid / softmax — Squash $\mathbb{R}$ to $(0,1)$ / probability simplex. Sigmoid for binary, softmax for $K$ classes.
• Cross-entropy loss — Information-theoretic distance between distributions. Categorical: $-{\sum }_k{y}_k\log {\hat{y}}_k$. Gradient on logits = $\hat{y}-y$ — clean and stable.
• Backpropagation — Chain-rule traversal of a computation graph from loss back to parameters. Each node multiplies the local Jacobian.
• Vanishing / exploding gradients — Gradient norm decays to 0 (or blows up) as it propagates back through many layers / time steps. Mitigations: careful init, ReLU/skip connections (forward), LSTM/clip (recurrent).
• Batch Normalisation — Normalise activations per mini-batch to zero-mean unit-variance; then learnable scale and shift. Faster training, less init-sensitive, slight regularisation. At inference: use running mean/var, NOT batch stats.
• Dropout — Stochastically zero $p$ fraction of activations during training. Acts as an ensemble + regulariser. Disable (or rescale) at inference.
• Bagging vs Boosting — Bagging: parallel, bootstrap samples, reduces variance (Random Forest). Boosting: sequential, reweight errors, reduces bias (AdaBoost; Viola-Jones face detection).
• Gaussian Mixture Model — $p(x)={\sum }_k{\pi }_k\mathcal{N}(x\mid {\mu }_k,{\Sigma }_k)$. Trained by EM. Soft clustering with covariance-shaped components.
• EM algorithm — E-step: compute responsibilities given current parameters. M-step: re-estimate parameters by weighted MLE. Monotonically improves log-likelihood; converges to a local maximum.
• LSTM — Gated recurrent unit with three gates (forget, input, output) and an additive cell-state path that prevents vanishing gradients (constant error carousel).
• Precision / Recall / F1 — P = TP/(TP+FP), R = TP/(TP+FN), F1 = 2PR/(P+R) (harmonic mean). Use precision when FP costly; recall when FN costly.
• AP / mAP — Average Precision = area under the PR curve for one class. mAP = mean over classes (or queries).
• ROC vs PR curve — ROC: TPR vs FPR. PR: Precision vs Recall. Prefer PR for imbalanced data with rare positives.
• kNN — Lazy learner; predict by majority vote among k nearest training points. Small k → variance; large k → bias. k usually odd.
• Normal equations — Closed-form least squares: $\hat{\beta }=(X^{\top }X{)}^{-1}{X}^{\top }y$. Fails when $X^{\top }X$ is singular (collinearity) or too large to invert.
• L1 vs L2 regularisation — L1 = sum of |w|; drives weights to exactly 0 → feature selection. L2 = sum of w²; shrinks smoothly, no exact zeros.
• PCA — Find orthogonal directions of max variance. Steps: centre → covariance → eigendecompose → keep top-k. ${\lambda }_i$ = variance along $v_i$.
• SVD — $X=U\Sigma V^{\top }$. Right singular vectors $V$ are PCA components. Numerically stable. Best rank-$r$ approximation in Frobenius norm (Eckart–Young).
• LoRA — Low-Rank Adaptation: $\Delta W=AB$ with $A\in {\mathbb{R}}^{d\times r},B\in {\mathbb{R}}^{r\times d}$, $r\ll d$. Cheap fine-tuning of huge models. Same spirit as SVD.
• k-means — Hard-assignment clustering. Init → assign → update → repeat. Issues: spherical bias, init-sensitive, outlier-sensitive. Fixes: k-means++, GMM, k-medoids.
• Hierarchical clustering — Agglomerative (bottom-up) or divisive (top-down). Produces a dendrogram; cut at different heights for different cluster counts. Linkage: single / complete / average / Ward.

Formulas

• $$\text{Sigmoid:}\sigma (z)=\frac{1}{1+e^{-z}};{\sigma }^{\prime }(z)=\sigma (z)(1-\sigma (z))$$
• $$\text{Softmax:}{p}_k=\frac{e^{w_k^{\top }x}}{{\sum }_j{e}^{w_j^{\top }x}}$$
• $$\text{Binary cross-entropy:}L=-y\log \hat{y}-(1-y)\log (1-\hat{y})$$
• $$\text{Categorical CE:}L=-\sum _k{y}_k\log {\hat{y}}_k$$
• $$\text{SGD step (logistic):}w\leftarrow w-\eta (\hat{y}-y)x$$
• $$\text{Kaiming (ReLU) init:}w\sim \mathcal{N}(0,2/n_{\text{in}})$$
• $$\text{BatchNorm:}\hat{x}=(x-{\mu }_B)/\sqrt{{\sigma }_B^2+ϵ};y=\gamma \hat{x}+\beta$$
• $$\text{LSTM cell:}{c}_t=f_t\odot c_{t-1}+i_t\odot {\tilde{c}}_t;h_t=o_t\odot \tanh (c_t)$$
• $$\text{Precision / Recall / F1:}P=\frac{TP}{TP+FP},R=\frac{TP}{TP+FN},F_1=\frac{2PR}{P+R}$$
• $$\text{Normal equations:}\hat{\beta }=(X^{\top }X{)}^{-1}{X}^{\top }y$$
• $$\text{PCA projection:}Z=\tilde{X}{V}_k\text{where}{V}_k=\text{top-}k\text{ eigenvectors of}\Sigma$$
• $$\text{SVD:}X=U\Sigma V^{\top };{\sigma }_i=\sqrt{{\lambda }_i};\text{best rank-}r\text{ approx (Eckart–Young)}$$
• $$\text{GMM EM E-step:}{\gamma }_{ik}=\frac{{\pi }_k\mathcal{N}(x_i\mid {\mu }_k,{\Sigma }_k)}{{\sum }_j{\pi }_j\mathcal{N}(x_i\mid {\mu }_j,{\Sigma }_j)}$$
• $$\text{k-means objective:}J=\sum _{i,k}1[c_i=k]\Vert x_i-{\mu }_k{\Vert }^2$$

Derivations

**Logistic regression SGD = $(\hat{y}-y)x$.** Loss $L=-y\log \hat{y}-(1-y)\log (1-\hat{y})$ with $\hat{y}=\sigma (w^{\top }x)$. Use chain rule: $\partial L/\partial w=(\partial L/\partial \hat{y})(\partial \hat{y}/\partial z)(\partial z/\partial w)$. $\partial L/\partial \hat{y}=-y/\hat{y}+(1-y)/(1-\hat{y})=(\hat{y}-y)/(\hat{y}(1-\hat{y}))$. $\partial \hat{y}/\partial z=\hat{y}(1-\hat{y})$. Cancellation gives $\partial L/\partial z=\hat{y}-y$, hence $\partial L/\partial w=(\hat{y}-y)x$. The cancellation is precisely why CE+sigmoid avoids vanishing-gradient saturation issues.

Why not zero-init. Suppose $W^{(1)}=0$. Then every neuron in layer 1 outputs the same activation ($\varphi (0)=$ same constant). Layer 2's gradients $\partial L/\partial W_{ij}^{(2)}$ depend only on the layer-1 output, which is the same for all $j$ → $W^{(2)}$ rows update identically → all neurons remain identical. By induction the network is equivalent to a single-neuron-per-layer network forever.

Bessel-style argument for cross-entropy gradient. With softmax + categorical CE, the gradient on logits is again ${\hat{y}}_k-y_k$ — no derivative-of-softmax factor survives. Gives stable, scale-invariant updates regardless of the magnitude of logits.

Normal equations. Minimise $J(\beta )=\Vert X\beta -y{\Vert }^2=(X\beta -y{)}^{\top }(X\beta -y)$. Expand: ${\beta }^{\top }{X}^{\top }X\beta -2y^{\top }X\beta +y^{\top }y$. Set $\partial J/\partial \beta =2X^{\top }X\beta -2X^{\top }y=0\Rightarrow \hat{\beta }=(X^{\top }X{)}^{-1}{X}^{\top }y$.

PCA via eigendecomposition. Project $\tilde{X}$ onto unit vector $v$: variance of projection = $v^{\top }\Sigma v$. Maximise subject to $\Vert v\Vert =1$ via Lagrangian: $\Sigma v=\lambda v$ — $v$ is an eigenvector, $\lambda$ is the variance (the corresponding eigenvalue). Top-$k$ eigenvectors maximise sequentially.

SVD low-rank optimum (Eckart–Young). For $X=U\Sigma V^{\top }$ with singular values ${\sigma }_1\ge {\sigma }_2\ge \dots$, the rank-$r$ matrix that minimises $\Vert X-X_r{\Vert }_F$ is $X_r=U_{:r}{\Sigma }_{:r}{V}_{:r}^{\top }$, with error $\sqrt{{\sigma }_{r+1}^2+\dots +{\sigma }_n^2}$. Same statement holds in the operator norm.

EM monotonically improves likelihood. E-step computes the variational lower bound $Q(\theta \mid {\theta }_t)$. M-step maximises $Q$. By Jensen's inequality, $\log p(X\mid {\theta }_{t+1})\ge Q({\theta }_{t+1}\mid {\theta }_t)\ge Q({\theta }_t\mid {\theta }_t)=\log p(X\mid {\theta }_t)$. Hence likelihood never decreases.

Examples

• Logistic regression decision boundary. Two clusters in 2D. After training, $w\approx (1,1)$, $b\approx -2$. Boundary: $x_1+x_2=2$ (a line). Points above predicted class 1, below class 0.
• Backprop on a 2-layer MLP. Compute output $\hat{y}=\sigma (W_2\varphi (W_{1}x+b_1)+b_2)$ on a 2D input by hand; then propagate gradient back through $W_2,W_1$. Practise this with $W_1=I$, $\varphi =\text{ReLU}$ — the canonical exam numerical.
• Confusion matrix. TP=80, FP=20, FN=10, TN=890. $P=80/100=0.80$. $R=80/90\approx 0.89$. $F_1=2\cdot 0.80\cdot 0.89/(0.80+0.89)\approx 0.84$.
• AP from a ranked list. 3 GTs, 5 ranked detections [TP, TP, FP, FP, TP]. After each step $(R,P)$ = $(0.33,1.0),(0.67,1.0),(0.67,0.67),(0.67,0.5),(1.0,0.6)$. AP ≈ 0.86 (area under staircase).
• PCA on toy 2D data. Centre, compute $2\times 2$ covariance, eigendecompose. Top eigenvector points along the data's main axis. Projection reduces 2D → 1D with the least variance loss.
• k-means iteration trace. Two clusters in 1D: data $\{1,2,3,10,11,12\}$. Init centres ${\mu }_0=0,{\mu }_1=5$. Assignment → $\{1,2,3\}\to 0$, $\{10,11,12\}\to 1$. Recompute → ${\mu }_0=2,{\mu }_1=11$. Converged.
• GMM E-step. Two Gaussians with ${\mu }_0=2,{\mu }_1=11$, $\sigma =1$, equal priors. For $x_i=6$: density under $k=0$ very small; under $k=1$ very small. Use ratio for $\gamma$ — both responsibilities ≈ 0.5 (point sits between modes).

Diagrams

• MLP block diagram: input → linear → activation → linear → softmax. Annotate forward and backward arrows.
• Sigmoid vs ReLU vs tanh activation curves with gradient curves underneath.
• LSTM cell schematic: three gates, the additive cell-state highway, output gate.
• Confusion matrix template (2×2 grid) with TP/FP/FN/TN labels.
• PR vs ROC curves on the same imbalanced-data toy problem to illustrate why PR is preferred.
• PCA: 2D cloud with two orthogonal arrows (principal axes) overlaid; rotated view = the projection.
• SVD diagram: $X=U\Sigma V^{\top }$ with $U$, diagonal $\Sigma$, $V^{\top }$ as labelled rectangles.
• k-means iteration: 3 panels (init, assign, update) on the same 2D point cloud.
• Dendrogram for hierarchical clustering on 6 points; cut at two different heights → 2 vs 3 clusters.

Edge cases

• MSE on logistic regression — non-convex with sigmoid; gradients vanish at saturation. Use cross-entropy.
• Zero-init kills the network via symmetry — all neurons identical forever.
• Dropout at inference must be turned OFF (or scaled by $1-p$ if using vanilla); inverted dropout scales during training so inference is unchanged.
• BatchNorm at inference uses running mean/var from training, NOT mini-batch statistics. Mismatch is a classic deployment bug.
• Curse of dimensionality breaks kNN and KDE in high $d$ — distances concentrate, nearest = farthest.
• LSTM exploding gradients are NOT solved by the gating — clip gradients separately.
• Imbalanced classification + accuracy is meaningless (99% accuracy = always say 'no cancer'). Use Precision/Recall/F1 or AUC-PR.
• Multinomial logistic / softmax is not the same as a stack of binary logistic regressions — softmax models mutual exclusivity, $K$ binaries don't.
• PCA assumes linear directions — fails on data lying on curved manifolds (use kernel PCA or t-SNE/UMAP).
• k-means with bad init can converge to a poor local minimum. Use k-means++ initialisation or run multiple restarts.
• Hierarchical clustering with single-linkage can produce 'chains' (long thin clusters); complete or average linkage are more compact.

Common mistakes

• Train/Val/Test confusion. Never tune on test. Val for hyperparams; test ONCE.
• MSE for classification. Use cross-entropy — gradient signal is granular and doesn't vanish.
• Accept H₀ vs fail to reject. (Not in this course but adjacent.) Hypothesis testing never 'accepts' the null.
• Confusing bagging and boosting. Bagging: parallel, independent → variance reduction (RF). Boosting: sequential, reweighting → bias reduction (AdaBoost, Viola-Jones).
• PCA without centring — the first PC will end up pointing at the mean, not the direction of variance. Always centre first.
• SVD vs PCA confusion. PCA = eigendecomposition of covariance. SVD = direct decomposition of $X$. Same answer, SVD is more numerically stable.
• Reporting accuracy on imbalanced data. Useless. Use Precision/Recall/F1 or AP/AUC-PR.
• Off-by-one in F1. $F_1=2PR/(P+R)$, NOT $(P+R)/2$ — that's the arithmetic mean. Harmonic mean punishes one bad value.
• k-means with k=1. Trivial. Always pick $k>1$ and use elbow method / silhouette score to choose.
• Polynomial regression is 'non-linear'. Wrong — linear in coefficients (you still solve normal equations). Truly non-linear models need iterative optimisation.

Shortcuts

• Logistic SGD: $w\leftarrow w-\eta (\hat{y}-y)x$.
• Why not MSE? Non-convex + vanishing gradient with sigmoid.
• Kaiming init for ReLU: factor of 2 to compensate for half-zeroing.
• Speed-up six: BN, higher LR, dropout, shuffle, ↓L2, LR decay.
• LSTM additive cell state = constant error carousel — solves vanishing gradient.
• Bagging reduces variance, Boosting reduces bias. RF = bagging. AdaBoost (Viola-Jones) = boosting.
• Precision when FP costly; Recall when FN costly.
• PR > ROC on imbalanced data.
• **$k$-means issues**: hard assignment, spherical bias, init-sensitive, outlier-sensitive. Fix: $k$-means++, GMM, $k$-medoids.
• PCA = eigenvectors of cov; SVD numerically stable; LoRA = low-rank update.

Proofs / Algorithms

**Cross-entropy gradient = $(\hat{y}-y)x$.** Shown above by chain rule; the cancellation between $\partial L/\partial \hat{y}$ and $\partial \hat{y}/\partial z$ produces a clean, scale-stable gradient — the reason CE is the canonical classification loss.

PCA solution is top eigenvectors. Variance along $v$ = $v^{\top }\Sigma v$; subject to $\Vert v\Vert =1$, Lagrangian gives $\Sigma v=\lambda v$ → eigen-equation. Successive directions found by deflating and repeating.

Eckart–Young. For $X=U\Sigma V^{\top }$, the best rank-$r$ approximation in Frobenius (or operator) norm is $X_r=U_{:r}{\Sigma }_{:r}{V}_{:r}^{\top }$, with error $\sqrt{{\sigma }_{r+1}^2+\dots }$.

EM monotonically improves likelihood. E-step computes tight lower bound; M-step maximises bound; therefore likelihood at ${\theta }_{t+1}$ ≥ likelihood at ${\theta }_t$ by Jensen's inequality.