Activation Functions

Sources:

• Understanding Categorical Cross-Entropy Loss, Binary Cross-Entropy Loss, Softmax Loss, Logistic Loss, Focal Loss and all those confusing names (Raúl Gómez blog)

Sigmoid

It transforms a vector in the continuous range [0, 1]. It's also called logistic function.

Formula

$$f(s_i) = \frac{1}{1 + e^{-s_i}}$$

Example in tf.keras

A sigmoid activation may be used within a Dense Keras Layer:

from tensorflow.keras.layers import Dense
from tensorflow.keras.models import Model

input_layer = # Previous layer

logits = Dense(
    units=32,
    activation='sigmoid'
)(input_layer)

model = Model(inputs=input_layer, outputs=logits)

Example in tf.math

It provides support to the operation by computing sigmod of x element-wise.

import tensorflow as tf

tf.math.sigmoid(
    x, # A tensor of "floating type"
    name=None # Optional
)

Softmax