Machine Learning — Visual Recall Sheet

Recall board 1 · Weeks 1–4

Reason first. Measure honestly.

Frame the answer, seal evaluation, then reason from geometry and counts. Every change below has a visible consequence.

DefineName the job. MechanismShow flow or math. PerturbMove one condition. ConcludeChoose + trade-off.

Notation + reasoning

Translate symbols into a story

\begin{matrix} \hat{y} = f_{θ} (x) \\ θ^{*} = \underset{θ}{\arg \min} L (θ) \end{matrix}

$x$: one input or feature vector
$f_{θ}$: model governed by parameters
$\hat{y}$: model prediction
$θ^{*}$: parameter choice with the smallest loss
$\underset{θ}{\arg \min}$: choose the parameter value that minimizes the following quantity

Parameter is learned; hyperparameter is chosen on validation evidence.
Representation is what numbers the learner sees; inductive bias is the solution shape it prefers.

Paradigms: supervised = labels · unsupervised = structure · semi-supervised = few labels + many unlabeled · reinforcement = reward. Bayes error is the irreducible floor.

Perturb: change representation → distances, splits, margins, and learned features can all change.

Evaluation + leakage

Train learns. Validation chooses. Test stays sealed.

Trainfit weights

Train onlyfit scaler

Validatechoose settings

Freezeall choices

Test oncefinal estimate

Deploywatch shift

Actual ↓Pred +Pred − Positive

TP

FN

2 Negative

FP

TN

Accuracy96%

Precision80%

Recall80%

F180%

\begin{matrix} P = \frac{TP}{TP + FP} R = \frac{TP}{TP + FN} \\ F_{1} = \frac{2 P R}{P + R} \\ specificity = \frac{TN}{TN + FP} \\ FPR = 1 - specificity \end{matrix}

$TP / TN$: correct positive / negative counts
$FP / FN$: incorrect positive / negative counts
$P / R$: precision / recall
$F_{1}$: harmonic mean of precision and recall
$specificity$: fraction of negatives correctly rejected
$FPR$: false-positive rate

CV: rotate $k$ validation folds; LOOCV uses $k = n$ . Fit scaler, preprocessing, and feature selection inside each training fold. ROC plots TPR vs FPR; PR plots precision vs recall.

Leakage: selecting depth, threshold, epoch, preprocessing, or features after looking at test results turns that “test” into validation. Obtain a new untouched test set.

Features + trees

Pick the split that removes weighted impurity

\begin{matrix} H (S) = - \sum_{c = 1}^{C} p_{c} \log_{2} p_{c} \\ IG = H (parent) - \sum_{v} \frac{| S_{v} |}{| S |} H (S_{v}) \end{matrix}

$S$: examples at the current node
$p_{c}$: fraction belonging to class
$C$: number of classes
$S_{v}$: child subset for branch
$| S |$: number of examples in that set
$IG$: information gained by the split

Micro-exampleParent: four positive / four negative, so

H = 1

. After a split, weighted

H \approx 0.451

, so

IG \approx 0.549

Features: preprocess → extract → select. Filter scores before a model; wrapper searches with CV. Fit both within folds; univariate filters miss XOR interactions.

Continuous candidates are adjacent-value midpoints. Weight every child; prune/stop depth for variance. Many-valued features inflate IG → use gain ratio.

KNN

Distance → nearest neighbors → vote

\begin{matrix} d_{p} (x, z) = {(\sum_{j = 1}^{d} {| x_{j} - z_{j} |}^{p})}^{\frac{1}{p}} \\ {\tilde{x}}_{j} = \frac{x_{j} - μ_{j, train}}{σ_{j, train}} \end{matrix}

$p$: norm order: Manhattan at $p = 1$ , Euclidean at $p = 2$
$d$: number of input features
$z$: comparison point whose distance is measured
${\tilde{x}}_{j}$: standardized value of one feature
$μ_{j, train} / σ_{j, train}$: training mean / standard deviation for that feature

Small $k$ : flexible, high variance. Large $k$ : smoother, more bias.
Scale first when units differ; choose $k$ and metric with validation/CV.
Weight by $\frac{1}{d^{p}}$ ; kd-trees help mainly in low dimensions; condensation stores fewer prototypes.

Perturb: scaling can reorder neighbors and flip the vote. If

d = 0

, use exact matches instead of

\frac{1}{d}

SVM

The closest points set the widest margin

\begin{matrix} \hat{y} = sign (w^{T} x + b) \\ margin width = \frac{2}{‖ w ‖} \\ \min_{w, b, ξ} \frac{1}{2} {‖ w ‖}^{2} + C \sum_{i = 1}^{n} ξ_{i} \\ subject to y_{i} (w^{T} x_{i} + b) \geq 1 - ξ_{i} \\ ξ_{i} \geq 0 \end{matrix}

$w$: boundary orientation and margin scale
$b$: boundary offset
$C$: penalty for margin violations
$ξ_{i}$: slack of one training case
$sign (\cdot)$: turns the score into the negative or positive class
$x_{i} / y_{i}$: training input / signed class target

Move a non-support point: boundary normally unchanged.
Move a support vector: optimum can move.
Lower $C$ : tolerate slack, usually wider margin. Higher $C$ : punish violations.
Polynomial/RBF kernels bend the boundary; large RBF $γ$ is local/spiky. One-vs-rest trains one classifier per class; one-vs-one trains one per class pair.

Diagnosis + metrics

Read the symptom before prescribing

Train high · val highunderfit → capacity/features

Train low · val highoverfit → regularize/data

Both good · deploy badshift or leakage

Lower threshold → more predicted positives: recall tends up, precision may fall.
Rare positives → inspect precision-vs-recall; its baseline is prevalence. ROC (TPR-vs-FPR) can look optimistic.
Macro weights classes equally; weighted follows support; micro pools decisions.

Constraint first: “catch ≥95%” means tune validation threshold for recall, then compare precision/workload among qualifying choices.

Test set: one final estimate after every decision is frozen. A zero metric denominator is undefined, not confidently zero. Error near the Bayes floor needs better information, not just more capacity.

Recall board 2 · Weeks 5–8

Trace signal forward. Trace blame backward.

Keep a shape ledger and a gradient ledger. Deep learning questions become short when every arrow names what it carries.

ShapeWrite dimensions. ForwardAffine then activate. LossCompare with target. BackwardLocal derivatives.

Neural forward + backprop

One graph, two directions

\begin{matrix} z_{h} = W_{h} x + b_{h}; h = ReLU (z_{h}) \\ z_{o} = v h + b_{o}; \hat{y} = σ (z_{o}) \\ L_{BCE} = - [y \ln \hat{y} + (1 - y) \ln (1 - \hat{y})] \\ \frac{\partial L}{\partial z_{o}} = \hat{y} - y \\ σ^{'} = σ (1 - σ); \tanh^{'} = 1 - \tanh^{2} \\ {ReLU}^{'} = 1 if z > 0, else 0 \end{matrix}

$z_{h} / z_{o}$: hidden / output pre-activation values
$h$: hidden activation passed to the output
$W_{h} / v$: hidden / output trainable weights
$b_{h} / b_{o}$: hidden / output biases
$σ / L_{BCE}$: sigmoid function / binary cross-entropy loss
$\frac{\partial L}{\partial z_{o}}$: loss sensitivity entering the output unit

NumbersUsing

x = [1, 2], W_{h} = [.5, .25]

and

b_{h} = 0, v = .8, b_{o} = - .3

gives

h = 1, z_{o} = .5, \hat{y} = .622

. For

y = 1

, the output gradient is

- .378

Sigmoid/tanh can saturate; ReLU can die (leaky ReLU keeps a small negative slope). Identical zero-initialized hidden units stay symmetric. Backprop follows each dependency backward.

Optimization + regularization

Optimization fits; regularization generalizes

\begin{matrix} v_{t} = β v_{t - 1} + g_{t}; θ_{t + 1} = θ_{t} - η v_{t} \\ L_{2} = L + λ_{2} \sum_{i} w_{i}^{2}; L_{1} = L + λ_{1} \sum_{i} | w_{i} | \\ elastic net = L + λ_{1} \sum_{i} | w_{i} | + λ_{2} \sum_{i} w_{i}^{2} \end{matrix}

$g_{t}$: current loss gradient
$v_{t}$: momentum accumulator
$β$: how much previous momentum is retained
$η$: learning rate or update step size
$λ_{1} / λ_{2}$: sparsity / shrinkage strengths

Batch=all; stochastic=1; mini-batch=middle. Adam rescales coordinates with gradient moments.
On a quadratic, $0 < η < \frac{2}{λ_{max}}$ ; tiny crawls, near-limit oscillates, larger diverges. Warm up, then decay.
$L_{1}$ zeros; $L_{2}$ shrinks; elastic net combines. Momentum smooths alternating gradients.
Augment only label-preserving transforms. Dropout off at inference; BatchNorm uses running, not batch, statistics.

Train loss falls while validation rises → restore best validation checkpoint; do not inspect test to choose the epoch.

CNN

Local window, shared weights, shape ledger

\begin{matrix} D = 1 : N_{out} = ⌊ \frac{N + 2 P - K}{S} ⌋ + 1 \\ parameters = (K_{h} K_{w} C_{in} + 1) C_{out} \end{matrix}

$D$: dilation; the displayed size rule assumes one
$N / N_{out}$: input / output side length
$K / S / P$: kernel width / stride / zero padding
$K_{h} / K_{w}$: kernel height / width
$C_{in} / C_{out}$: input / output channel counts
$+ 1$: one learned bias per output filter

Example

32 \times 32 \times 3, K = 3, S = 1, P = 1, 16 filters

gives

32 \times 32 \times 16 and (3 \cdot 3 \cdot 3 + 1) 16 = 448

parameters.

Conv32×32×16

Pool 2×216×16×16

Flatten4096

Dense $m$

(4096 + 1) m

“Same” at

S = 1

keeps size with

P = \frac{K - 1}{2}

; “valid” means

P = 0

. Keep the floor. Pooling has zero learned parameters.

RNN + LSTM

A recurrent path can fade; a gated highway can carry

\begin{matrix} h_{t} = \tanh (W_{x} x_{t} + W_{h} h_{t - 1} + b) \\ c_{t} = f_{t} ⊙ c_{t - 1} + i_{t} ⊙ g_{t} \\ h_{t} = o_{t} ⊙ \tanh (c_{t}) \end{matrix}

$x_{t}$: input vector at the current sequence step
$h_{t}$: hidden state exposed to later computation
$c_{t}$: LSTM memory or cell state
$W_{x} / W_{h}$: input / recurrent weight matrices
$f_{t} / i_{t} / g_{t} / o_{t}$: forget / input / candidate / output gate vectors
$⊙$: element-by-element multiplication

Forget $f$ keep old memory near

1

Input $i$ write candidate near

1

Output $o$ reveal state near

1

Numbers

c_{prev} = .8, f = .75, i = .4, g = .5

gives

c = .6 + .2 = .8

. With

o = .9

h \approx .598

GRU merges forget/input into an update gate and has no separate cell state. Bidirectional needs the whole sequence—invalid for strict streaming. Seq2seq attention selects encoder states instead of one fixed summary.

γ < 1

across many steps → vanishing: gates/attention.

γ > 1

→ exploding: clip gradients.

Transformers

Match with $Q \cdot K$ ; move content with $V$

\begin{matrix} A = {softmax}_{row} (\frac{Q K^{T}}{\sqrt{d_{k}}} + M); O = A V \\ M_{i, j} = 0 if allowed; -\infty if forbidden \end{matrix}

$Q / K / V$: query / key / value matrices
$d_{k}$: key-vector width used to scale scores
$M_{i, j}$: mask entry for query row and key column
$A / O$: attention-weight matrix / resulting output
${softmax}_{row}$: normalize each score row to sum to one

Inputtoken embedding + position

Multi-headparallel learned

Q / K / V

projections

Blockattention → Add&Norm → FFN → Add&Norm

Encoder-only: bidirectional (BERT: masked language modeling + original next-sentence objective).
Decoder-only: causal generation; teacher forcing during training, autoregressive feedback at inference.
Encoder–decoder: decoder $Q$ cross-attends to encoder $K / V$ ; useful for sequence-to-sequence tasks.

Mask before softmax. Zeroing probabilities afterward breaks row sums and does not renormalize the allowed evidence.

Architecture routing

Choose the information path

CNNlocal, repeated spatial patterns; shared filters.

RNNonline sequence, compressed past; long gradient path.

LSTMsequential, gated long memory; still cannot parallelize time.

Transformerdirect token links, parallel training; standard full attention costs

O (n^{2})

Exam verb “compare” → path length, parallelism, memory mechanism, and cost. Do not answer with popularity.

Softmax + categorical CE gives $\nabla_{z} L = p - y$ . Adding the same constant to all logits leaves probabilities unchanged.

Recall board 3 · Weeks 9–10

Route by constraint. Verify by evidence.

Choose whether to change instructions, context, a small update, all weights, or the representation—and isolate the exact failed stage.

ConstraintFreshness, labels, GPU. InterventionWhat actually changes? TestIsolate one stage. EvidenceAgreement is not truth.

Method selector

Change the smallest thing that solves the constraint

Promptknown task; format or instruction steering

RAGfresh/private facts; citations; no retraining

PEFTstable labeled task; limited memory

Full tunelarge stable corpus; compute; deep change

Prompt: zero-shot → few-shot examples → explicit reasoning steps when justified. Transfer: frozen feature extractor → partial → full tune. PEFT: soft prompt, per-layer prefix, adapters, or mergeable LoRA. Pruning removes low-value weights/units after validation.

Weekly policy updates + strict citations → RAG. A response-format failure with known knowledge → prompt/template first.

Exam routing

Turn the question cue into a complete answer

“Calculate”write formula → substitute → units/shape → sanity check

“Explain why”mechanism → consequence → one perturbation

“Compare”same axes for both methods → scenario choice

“Diagnose”observed symptom → isolating test → stage-specific fix

Fast routeMetric cost → threshold. Weighted counts → tree. Geometry → KNN/SVM. Shape → CNN. Long dependency → LSTM/attention. Fresh evidence → RAG. Limited adaptation budget → LoRA. Domain shift → DANN.

Never list every method. State the constraint, select one intervention, name what changes, and pay one honest trade-off.

RAG: six separate failure stages

Wrong answer is a symptom, not a diagnosis

1 Corpusevidence exists?

2 Chunkfact stays whole?

3 Retrieveenters top-k?

4 Reranksurvives cutoff?

5 Generateuses evidence?

6 Citepoints correctly?

1 · CoverageAbsent from owned corpus → acquire/curate source.

2 · ChunkingFact split across chunks → semantic boundary/overlap.

3 · RetrievalWhole gold chunk misses top-k → query/embedding/recall.

4 · RankingGold is rank 18 and cut → rerank shortlist.

5 · GroundingGold is in prompt; answer contradicts → generation constraint.

6 · AttributionAnswer supported; citation wrong → claim-to-passage check.

Offline → onlineCurate/chunk/embed/index once → embed query → fast bi-encoder retrieve → precise cross-encoder rerank shortlist → prompt generator → verify claim-to-passage.

Gold-passage test: inject the correct passage manually. Answer fixed → stages 1–4. Still wrong → stage 5. Correct claim, wrong pointer → stage 6.

LoRA

Freeze $W_{0}$ ; learn a low-rank update

\begin{matrix} ΔW = B A \\ trainable = r (d_{in} + d_{out}) \\ square d \times d special case = 2 d r \end{matrix}

$W_{0}$: frozen pretrained weight matrix
$ΔW$: learned low-rank update
$A / B$: small trainable factor matrices
$r$: chosen rank or bottleneck width
$d_{in} / d_{out}$: input / output dimensions of the base matrix

Example

d_{in} = d_{out} = 4096, r = 8

gives

65,536 trainable vs 16,777,216 full = 0.391 %

Initialize

B = 0

ΔW = 0

at start; merge

W_{0} + B A

for zero added inference latency. Use

r (d_{in} + d_{out})

;

2 d r

is only square.

Verification

Ask: what independent evidence enters?

Rulesdeterministic, precise, narrow

Retrievalexternal trusted evidence; retrieval is gate

LLM judgeflexible rubric; shared-bias risk

Reflectioncheap revision; cannot create missing evidence

Consistencyaggregate paths; more compute

Multi-agentseparate roles; correlated failures remain

Agreement is not truth. Internal checks become factual verification only when claims are compared with trusted evidence.

DANN

Keep label signal; erase domain signal

\nabla_{θ_{f}} L_{d} \mapsto - λ \nabla_{θ_{f}} L_{d}

$θ_{f}$: feature-extractor parameters
$L_{d}$: domain-classification loss
$λ$: strength of the reversed domain signal
$\nabla_{θ_{f}}$: gradient with respect to extractor parameters
$\mapsto$: replace the incoming gradient during backpropagation

Example: labeled hospital A, unlabeled hospital B. Preserve disease evidence while suppressing hospital/scanner cues.

Assumption: source and target share a useful labeling structure. Domain confusion alone must not erase task signal.

Final recall check

Ten checks before you turn the page

Did I name the decision and trade-off?
Is every learned transform train-only?
Did I weight tree children?
Did I scale distance features?
Which points actually set the margin?
Did I preserve tensor dimensions?
Where can the gradient vanish?
Is the causal mask before softmax?
Did I count LoRA as $r (d_{in} + d_{out})$ ?
Which exact RAG stage failed?

Reconstruct: close the page and redraw the sealed evaluation flow, forward/backward arrows, masked attention, and six RAG stages. Reopen and mark only omissions.

Bounded to Weeks 1–10; this recall layer does not replace the guide’s worked labs, drills, flashcards, or mock exam. Complete guide →

Back to first board ↑

Machine Learning visual recall sheet

Translate symbols into a story

Train learns. Validation chooses. Test stays sealed.

Pick the split that removes weighted impurity

Distance → nearest neighbors → vote

The closest points set the widest margin

Read the symptom before prescribing

One graph, two directions

Optimization fits; regularization generalizes

Local window, shared weights, shape ledger

A recurrent path can fade; a gated highway can carry

Match with Q·K; move content with V

Choose the information path

Change the smallest thing that solves the constraint

Turn the question cue into a complete answer

Wrong answer is a symptom, not a diagnosis

Freeze W0; learn a low-rank update

Ask: what independent evidence enters?

Keep label signal; erase domain signal

Ten checks before you turn the page

Match with $Q \cdot K$ ; move content with $V$

Freeze $W_{0}$ ; learn a low-rank update