Basics

Basic Derivatives

Function	Derivative
$f(x) = C$ (constant)	$f'(x) = 0$
$f(x) = x^n$	$f'(x) = nx^{n-1}$
$f(x) = 1-exp(Cx)$	$f'(x) = -C \exp(Cx)$
$f(x) = \exp(x)$	$f'(x) = \exp(x)$
$f(x) = \exp(f(x))$	$f'(x) = \exp(f(x)) \cdot f'(x)$
$f(x) = \log(x)$	$f'(x) = \frac{1}{x}$
$f(x) = \log(C)$	$f'(x) = 0$
$f(x) = g(h(x))$	$f'(x) = g'(h(x)) \cdot h'(x)$
$f(x) = u(x) \cdot v(x)$	$f'(x) = u'(x) \cdot v(x) + u(x) \cdot v'(x)$
$f(x) = \lvert x \rvert$	$f'(x) = sign(1)$

Common ML Functions

• Logarithm and Exponential:

  Logarithm Rules	Exponential Rules
  $\ln(ab) = \ln(a) + \ln(b)$	$e^a \cdot e^b = e^{a+b}$
  $\ln\left(\frac{a}{b}\right) = \ln(a) - \ln(b)$	$\frac{e^a}{e^b} = e^{a-b}$
  $\ln(a^b) = b \ln(a)$	$(e^a)^b = e^{ab}$
  $\ln(e) = 1$	$e^0 = 1$
  $\ln(1) = 0$	$e^{\ln(x)} = x$
  $\ln(e^x) = x$	$e^{\ln(a) + \ln(b)} = ab$
  $\ln\left(\prod_{i} a_i\right) = \sum_{i} \ln(a_i)$	$\prod_{i} e^{f(x_i)} = e^{\sum_{i} f(x_i)}$

• Sigmoid Function:

$$\sigma(x) = \frac{1}{1 + e^{-x}}$$

$$\sigma'(x) = \sigma(x)(1 - \sigma(x))$$

• Softmax Function (for class $i$):

$$\text{softmax}(x_i) = \frac{e^{x_i}}{\sum_{j} e^{x_j}}$$

• Log-Likelihood:

$$\frac{d}{dx}\ln(f(x)) = \frac{f'(x)}{f(x)}$$

Gradient

The gradient is a vector of all partial derivatives:

$$\nabla f = \begin{bmatrix} \frac{\partial f}{\partial x_1} \\ \frac{\partial f}{\partial x_2} \\ \vdots \\ \frac{\partial f}{\partial x_n} \end{bmatrix}$$

The gradient points in the direction of steepest ascent, which is why gradient descent moves in the negative gradient direction to minimize loss functions.