Metrics

Sources:

• Understanding confusion matrix (Sarang Narkhede)

• Accuracy, Precision, Recall or F1? (Koo Ping Shung)

• Cost Matrix (IBM Knowledge Center)

In regression

TODO

In classification

The confusion matrix

Confusion Matrix is a performance measurement for machine learning classification where output can be two or more classes.

• True positive

  • Predicted 1 $\Rightarrow$ Actual 1

• False positive

  • Predicted 1 $\Rightarrow$ Actual 0

  • This is a Type-1 Error

• False negative

  • Predicted 0 $\Rightarrow$ Actual 1

  • This is a Type-2 Error

• True negative

  • Predicted 0 $\Rightarrow$ Actual 0

Cost matrix

A cost matrix (error matrix) is also useful when specific classification errors are more severe than others. The Classification mining function tries to avoid classification errors with a high error weight. The trade-off of avoiding 'expensive' classification errors is an increased number of 'cheap' classification errors.

Accuracy

From all the total samples, it measures the well-classified rate.

Formula

$$Accuracy = \frac{\text{TP} + \text{TN} }{Total}$$

Example with sklearn

It may be computed using scikit-learn:

from sklearn.metrics import accuracy_score

y_pred = [0, 2, 1, 3]
y_true = [0, 1, 2, 3]
accuracy_score(y_true, y_pred)
>>> 0.5

Precision

From all the predicted positives, it measures the rate actually classified as positive.

It is a good measure to determine, when the costs of FP is high. For instance, email spam detection.

Main

Formula

$$Precision = \frac{\text{TP}}{\text{TP} + \text{FP}}$$

Example with sklearn

It may be computed using scikit-learn:

from sklearn.metrics import precision_score

precision_score(y_true, y_pred, average='weighted')

Recall

From all the actual positives, it measures the rate classified as positives values. As well. it is called Sensitivity or True Positive Rate (TPR).

It is a good metric to select our best model when there is a high cost associated with FN. For instance, in fraud detection or sick patient detection.

Main

Formula

$$Recall = \text{TPR } = Sensitivity = \frac{\text{TP}}{\text{TP} + \text{FN}}$$

Example with sklearn

It may be computed using scikit-learn:

from sklearn.metrics import recall_score

recall_score(y_true, y_pred, average='weighted')

Minimizing False Positives

Specificity

From all the actual negative values, it measures the rate classified as negative.

$$Specificity = \frac{\text{TN}}{\text{TN} + \text{FP}}$$

False Positive Rate (FPR)

From all the actual negative values, it measures the rate misclassified as negative. It the negated probability of specificity.

$$\text{FPR} = 1-Specificity = \frac{\text{FP}}{\text{TN} + \text{FP}}$$

F-Score

F1-score might be a better measure to use if we need to seek a balance between Precision and Recall AND there is an uneven class distribution (large number of Actual Negatives).

Formula

$$\text{F1} = 2 \cdot \frac{Precision \cdot Recall}{Precision + Recall}$$

Example with sklearn

from sklearn.metrics import f1_score

f1_score(y_true, y_pred, average='weighted')