## Machine Learning: Core Formulas

A one-page reference of the equations behind training a neural network.

A classifier maps an input vector x to class probabilities. The raw scores (logits) z are turned into a distribution with the softmax function, where the probability of class i is

$$
\sigma ( z ) _ { i } = \frac { \mathbf { e } ^ { z _ { i } } } { \sum _ { j = 1 } ^ { K } \mathrm { e } ^ { z _ { j } } }
$$

## Loss

Training minimizes the cross-entropy loss between the predicted distribution and the one-hot target y:

$$
\mathcal { L } = - \frac { 1 } { n } \sum _ { i = 1 } ^ { n } \sum _ { c = 1 } ^ { K } y _ { i , c } \cdot \log \hat { Y } _ { i , c }
$$

## Gradient descent

Parameters θ are updated against the gradient of the loss, scaled by the learning rate η:

$$
\theta _ { t + 1 } = \theta _ { t } - \eta \nabla _ { \theta } \mathcal { L } ( \theta _ { t } )
$$

Mean squared error, used for regression, is $\mathrm { M S E } = ( 1 / n ) \sum ( y _ { i } - \hat { y } _ { i } ) ^ { 2 }$

Typical hyperparameters
<table><tr><td rowspan=1 colspan=1>Symbol</td><td rowspan=1 colspan=1>Name</td><td rowspan=1 colspan=1>Value</td></tr><tr><td rowspan=1 colspan=1>=</td><td rowspan=1 colspan=1>Learning rate</td><td rowspan=1 colspan=1> $3 \times { 1 0 } ^ { - 4 }$ </td></tr><tr><td rowspan=1 colspan=1>B</td><td rowspan=1 colspan=1>Batch size</td><td rowspan=1 colspan=1>32</td></tr><tr><td rowspan=1 colspan=1>λ</td><td rowspan=1 colspan=1>Weight decay</td><td rowspan=1 colspan=1>0.01</td></tr></table>

Sample document · equations rendered as vector text to demonstrate formula extraction.