Activation Functions

Sources:

• Understanding Categorical Cross-Entropy Loss, Binary Cross-Entropy Loss, Softmax Loss, Logistic Loss, Focal Loss and all those confusing names (Raúl Gómez blog)

Linear

The input equals the output $f(z) = z$. Meaning the output is continuous.

Sigmoid or Logistic function

It transforms a vector in the continuous range [0, 1].

Formula

$$f(s_i) = \frac{1}{1 + e^{-s_i}}$$

Example in tf.keras

Example in tf.math

Softmax

Converts its input values into a probability distribution among N classes.

Transforms raw scores into probabilities in a multi-label classification. All the probabilities are normalized to sum 1 all of them.

$$f(s_i) = \frac{ e^{x_i}}{\sum_j e^{x_j}}$$

ReLU

Also called Rectified Linear Unit,

• if the input value is ≥ 0, (acts as a Linear function)

• it the input value is < 0, it returns 0

It can be summarized as $ReLU(z) = max(0, z)$