Section outline
-
In computer vision, the “confusion matrix” is widely used, and it allows us to derive many relevant statistics to evaluate a model.
We need to go through a theoretical section before fully understanding what we are seeing, so you may need to focus a bit!
For this theoretical part, we will consider the case where we are trying to detect British coins called “pennies”.
CONFUSION MATRIX
A confusion matrix is a table that synthesizes and categorizes the model's predictions to facilitate their analysis.
Here, a distinction must be made between what is true and what was predicted. An object belongs to a specific class; this is referred to as the "ground truth" (e.g., a cat belongs to the "cat" class). Conversely, the model makes predictions by assigning a class to an object. However, the model can err; its prediction does not necessarily correspond to reality (for instance, a cat might be classified as a "dog").
Therefore, we distinguish between:
- ground truth : the actual class of the objects.
- predictions : the results generated by the model.
Within each of these two categories, there are two possible outcomes : positive (presence of the target object, e.g., a penny) and negative (absence of the target object).
This creates four possible combinations:
For easier interpretation, these four combinations are consolidated into a two-way table. The rows correspond to the model's predictions, while the columns represent the ground truth.
- TP: True Positive
- What should be detected and is correctly detected.
- Example: a “penny” is present and correctly detected.
- TN: True Negative
- What should not be detected and is not detected.
- Example: no “penny” is present and none is detected.
- FP: False Positive
- What should be negative but is predicted as positive.
- Example: no “penny” is present but one is detected.
- FN: False Negative
- What should be positive but is predicted as negative.
- Example: a “penny” is present but not detected.
EXAMPLE
The image below corresponds to the ground truth, i.e., the manually annotated data that the model is expected to detect:
The next image corresponds to the detections made by the model on the same image after training:
If we compare the two images:
We then fill in the confusion matrix with each instance:
Each cell is filled by counting the number of occurrences of each event. In this simplified example, there is one of each. True negatives are not counted in detection tasks, as they are not meaningful: saying that nothing was detected when nothing was present becomes noise at the image scale.
From this matrix, we can derive several metrics to evaluate the model. Ultralytics provides four curves: precision-confidence, recall-confidence, precision-recall, and F1 score. We will see how to interpret them, but first let’s define each concept.
CONFIDENCE
When the model makes a prediction, it outputs a value representing the certainty of the prediction, i.e., a probability. This value, between 0 and 1, is called confidence (1 meaning absolute certainty).
A threshold value must be chosen. If the confidence is above this threshold, the detection is accepted (labeled “penny”); otherwise, it is rejected (labeled “background”/negative).
EXAMPLE
Reading the graph: point “1” corresponds to prediction on image 1. It is green, so a “penny” is present. The model predicts it with a confidence of 0.55.
Point “2” corresponds to image 2. It is red (no “penny”), but the model predicts a “penny” with 0.6 confidence.
Let’s choose a threshold of 0.5. Points above are accepted, below are rejected :
How to fill the matrix :
-
Point “1” is above the confidence threshold, so it is predicted positive (contains a penny) and is green (actually contains a penny) → True Positive
-
Point “2” is above the confidence threshold, so it is predicted positive (contains a penny) and is red (does not contain a penny) → False Positive
-
Point “3” is below the confidence threshold, so it is predicted negative (does not contain a penny) and is green (actually contains a penny) → False Negative
-
Point “4” is below the confidence threshold, so it is predicted negative (does not contain a penny) and is red (does not contain a penny) → True Negative
-
Point “5” is above the confidence threshold, so it is predicted positive (contains a penny) and is green (actually contains a penny) → True Positive
Now let’s try a threshold of 0.75 :
-
Point “1” is below the threshold, so it is predicted negative but green → False Negative
-
Point “2” is below the threshold, so it is predicted negative and red → True Negative
-
Point “3” is below the threshold, so it is predicted negative but green → False Negative
-
Point “4” is below the threshold, so it is predicted negative and red → True Negative
-
Point “5” is above the threshold, so it is predicted positive and green → True Positive
Finally, let’s try a threshold of 0.35 :
- Point “1” is above the threshold, so it is predicted positive and is green → True Positive
- Point “2” is above the threshold, so it is predicted positive but is red → False Positive
- Point “3” is above the threshold, so it is predicted positive and is green → True Positive
- Point “4” is below the threshold, so it is predicted negative and is red → True Negative
- Point “5” is above the threshold, so it is predicted positive and is green → True Positive
For a given model, there are multiple possible confusion matrices, depending on the chosen confidence threshold :
In theory, there are an infinite number of confusion matrices, as there are an infinite number of threshold values between 0 and 1.
The confusion matrix reflects the quality of the chosen confidence threshold. One key objective after training is to select the best threshold for deployment.
But how to know if a confusion matrix is a good one. A good matrix maximizes the correct diagonal (true positives and true negatives) and minimizes the bad diagonal (false positives and falses negatives).
However, in practice, reducing both errors simultaneously is difficult, so trade-offs must be made. For example, in medicine, false negatives are minimized (to avoid missing a disease), even if it increases false positives.
To determine the best threshold, we rely on evaluation metrics which we gonna see in the next section.