Gaussian Mixture Model

Sources:

• Gaussian Mixture Models Explained (Towards Data Science)

• Gaussian Mixture Models Algorithm Explained (Cory Maklin)

What is a Gaussian Mixture?

It is function composed by several Gaussians. The number of Gaussians is equal to the number of clusters (k). Each distribution is parametrized by:

• The mean $\mu$ which defines its centre.

• The covariance Σ which defines its width.

• The mixing probability $\pi$.

The Gaussian density function is given by:

where:

• The sum of every mixing probability must be equal one: $\sum_{i=1}^K \pi_i = 1$

• $D$is the number of dimensions or features of each instance.

• Each data point $X$represents a data point (a $1 \times D$ vector)

• The mean $\mu$ is a $1 \times D$ vector.

• And the covariance Σ is a $D \times D$ matrix.

How do we fit the algorithm?

By applying the Expectation-Maximization algorithm widely used for optimization problems where the objective function is that complex.

Differences regarding k-Means

• It accounts for covariance, which determines the shape of the distribution This means that meanwhile the k-means model is that it places a circle (or a hyper-sphere) at the center of each cluster, a GMM model can handle different shapes.

• k-Means performs a hard classification, but a GMM model carries out a soft one by returning the probability that each data point belongs to a certain cluster.