Anomaly detection methods

Sources & credit:

• What machine learning technique is usually used to solve anomaly detection? (Quora)

• How to use machine learning for anomaly detection and condition monitoring (Vegard Flovik)

• 5 Ways to Detect Outliers/Anomalies That Every Data Scientist Should Know (Will Badr)

• A Brief Overview of Outlier Detection Techniques (Sergio Santoyo)

• Intuitively Understanding Variational Autoencoders (Irhum Shafkat)

• DBSCAN: What is it? When to use it? How to use it?

• Best clustering algorithms for anomaly detection

Depending on the presence or lack of labels, there are two approaches to face yourself to an anomaly detection problem.

As an unbalanced classification problem

When we are given a set of observations with labels that indicate whether each point is an anomaly or not, this can be seen as a binary classification problem. So we can use any classifier we like. The only issue here is that anomalies are by definition rare events, so you’ll have to deal with class imbalance.

How to deal with imbalanced datasets?

As an unsupervised problem

In this scenario, we are given a set of points without class labels. Some of them are anomalies and some aren’t, but you don’t know which is which. The goal here is to operationalize the intuitive idea that anomalies are different from the typical data point.

Outliers

Under construction

Basic methods

Z-Score

Z-score

IQR

IQR

Multivariate anomaly detection

Clustering

DBScan clustering

DBSCAN

Other clustering techniques

These clustering techniques may be used to detect instances that are far away from clusters.

kMeansGaussian Mixture Model

Tree-based approach

Isolation Forest

It is an unsupervised learning algorithm that belongs to the ensemble decision trees family. It explicitly isolates anomalies instead of profiling and constructing normal points and regions by assigning a score to each data point.

This algorithm works great with very high dimensional datasets and it proved to be a very effective way of detecting anomalies.

Robust Random Cut Forest (RCF)