Activation Functions

Sources:

• Understanding Categorical Cross-Entropy Loss, Binary Cross-Entropy Loss, Softmax Loss, Logistic Loss, Focal Loss and all those confusing names (Raúl Gómez blog)

Linear

The input equals the output $f(z) = z$. Meaning the output is continuous.

Sigmoid or Logistic function

It transforms a vector in the continuous range [0, 1].

Formula

$$f(s_i) = \frac{1}{1 + e^{-s_i}}$$

Example in tf.keras

A sigmoid activation may be used within a Dense Keras Layer:

from tensorflow.keras.layers import Dense
from tensorflow.keras.models import Model

input_layer = # Previous layer

logits = Dense(
    units=32,
    activation='sigmoid'
)(input_layer)

model = Model(inputs=input_layer, outputs=logits)

Example in tf.math

It provides support to the operation by computing sigmod of x element-wise.

import tensorflow as tf

tf.math.sigmoid(
    x, # A tensor of "floating type"
    name=None # Optional
)

Softmax

Converts its input values into a probability distribution among N classes.

Transforms raw scores into probabilities in a multi-label classification. All the probabilities are normalized to sum 1 all of them.

$$f(s_i) = \frac{ e^{x_i}}{\sum_j e^{x_j}}$$

ReLU

Also called Rectified Linear Unit,

• if the input value is ≥ 0, (acts as a Linear function)

• it the input value is < 0, it returns 0

It can be summarized as $ReLU(z) = max(0, z)$