RESULTS

Upon completion of training, the results are stored in the `runs/detect/train` directory, unless a custom path has been configured. This folder contains all the artifacts necessary for evaluating the model's performance and for its subsequent deployment.

Although this section details the complete interpretation of the generated data, a summary analysis of the success criteria is available directly in the conclusion of this chapter. The directory tree generally includes the following elements (subject to slight variations depending on the configuration):

The “Weights” folder corresponds to the result of your work, as it contains the weights of your model. It is essentially your model stored in a file.

There are two files inside: “best.pt” and “last.pt”. This can be understood as follows: the weights of the neurons change continuously during training, and the quality of these weights is evaluated using a specific function. Therefore, last.pt corresponds to the most recent weights, while best.pt corresponds to the best weights obtained throughout the entire training process. We will discuss later why having two files can be useful.

The `args.yaml` file contains all the parameters YOLO utilized for its training, including those with default values that you did not modify and those without arguments (thus set to 0).

Important point : verify that ".toml" is indeed the extension of your file. To do this, you must first enable the display of file extensions (View --> Show (at the very bottom) --> File name extensions), then ensure that the filename ends with ".toml".

In computer vision, the “confusion matrix” is widely used, and it allows us to derive many relevant statistics to evaluate a model.

We need to go through a theoretical section before fully understanding what we are seeing, so you may need to focus a bit!

For this theoretical part, we will consider the case where we are trying to detect British coins called “pennies”.

CONFUSION MATRIX

A confusion matrix is a table that synthesizes and categorizes the model's predictions to facilitate their analysis.

Here, a distinction must be made between what is true and what was predicted. An object belongs to a specific class; this is referred to as the "ground truth" (e.g., a cat belongs to the "cat" class). Conversely, the model makes predictions by assigning a class to an object. However, the model can err; its prediction does not necessarily correspond to reality (for instance, a cat might be classified as a "dog").

Therefore, we distinguish between:

• ground truth : the actual class of the objects.
• predictions : the results generated by the model.

Within each of these two categories, there are two possible outcomes : positive (presence of the target object, e.g., a penny) and negative (absence of the target object).

This creates four possible combinations:

For easier interpretation, these four combinations are consolidated into a two-way table. The rows correspond to the model's predictions, while the columns represent the ground truth.

• TP: True Positive
  • What should be detected and is correctly detected.
  • Example: a “penny” is present and correctly detected.

• TN: True Negative
  • What should not be detected and is not detected.
  • Example: no “penny” is present and none is detected.

• FP: False Positive
  • What should be negative but is predicted as positive.
  • Example: no “penny” is present but one is detected.

• FN: False Negative
  • What should be positive but is predicted as negative.
  • Example: a “penny” is present but not detected.

EXAMPLE

The image below corresponds to the ground truth, i.e., the manually annotated data that the model is expected to detect:

The next image corresponds to the detections made by the model on the same image after training:

If we compare the two images:

We then fill in the confusion matrix with each instance:

Each cell is filled by counting the number of occurrences of each event. In this simplified example, there is one of each. True negatives are not counted in detection tasks, as they are not meaningful: saying that nothing was detected when nothing was present becomes noise at the image scale.

From this matrix, we can derive several metrics to evaluate the model. Ultralytics provides four curves: precision-confidence, recall-confidence, precision-recall, and F1 score. We will see how to interpret them, but first let’s define each concept.

CONFIDENCE

When the model makes a prediction, it outputs a value representing the certainty of the prediction, i.e., a probability. This value, between 0 and 1, is called confidence (1 meaning absolute certainty).

A threshold value must be chosen. If the confidence is above this threshold, the detection is accepted (labeled “penny”); otherwise, it is rejected (labeled “background”/negative).

EXAMPLE

Reading the graph: point “1” corresponds to prediction on image 1. It is green, so a “penny” is present. The model predicts it with a confidence of 0.55.

Point “2” corresponds to image 2. It is red (no “penny”), but the model predicts a “penny” with 0.6 confidence.

Let’s choose a threshold of 0.5. Points above are accepted, below are rejected :

How to fill the matrix :

• Point “1” is above the confidence threshold, so it is predicted positive (contains a penny) and is green (actually contains a penny) → True Positive 

• Point “2” is above the confidence threshold, so it is predicted positive (contains a penny) and is red (does not contain a penny) → False Positive 

• Point “3” is below the confidence threshold, so it is predicted negative (does not contain a penny) and is green (actually contains a penny) → False Negative

• Point “4” is below the confidence threshold, so it is predicted negative (does not contain a penny) and is red (does not contain a penny) → True Negative

• Point “5” is above the confidence threshold, so it is predicted positive (contains a penny) and is green (actually contains a penny) → True Positive

Now let’s try a threshold of 0.75 :

• Point “1” is below the threshold, so it is predicted negative but green → False Negative 

• Point “2” is below the threshold, so it is predicted negative and red → True Negative

• Point “3” is below the threshold, so it is predicted negative but green → False Negative

• Point “4” is below the threshold, so it is predicted negative and red → True Negative 

• Point “5” is above the threshold, so it is predicted positive and green → True Positive 

Finally, let’s try a threshold of 0.35 :

• Point “1” is above the threshold, so it is predicted positive and is green → True Positive
• Point “2” is above the threshold, so it is predicted positive but is red → False Positive
• Point “3” is above the threshold, so it is predicted positive and is green → True Positive
• Point “4” is below the threshold, so it is predicted negative and is red → True Negative
• Point “5” is above the threshold, so it is predicted positive and is green → True Positive

For a given model, there are multiple possible confusion matrices, depending on the chosen confidence threshold :

In theory, there are an infinite number of confusion matrices, as there are an infinite number of threshold values between 0 and 1.

The confusion matrix reflects the quality of the chosen confidence threshold. One key objective after training is to select the best threshold for deployment.

But how to know if a confusion matrix is a good one. A good matrix maximizes the correct diagonal (true positives and true negatives) and minimizes the bad diagonal (false positives and falses negatives).

However, in practice, reducing both errors simultaneously is difficult, so trade-offs must be made. For example, in medicine, false negatives are minimized (to avoid missing a disease), even if it increases false positives.

To determine the best threshold, we rely on evaluation metrics which we gonna see in the next section.

Let’s visualize these metrics using a diagram:

Using the previous diagram, we can express precision and recall as follows:

Mathematically, this can be formalized as follows:

It is therefore possible to plot curves to observe how these metrics evolve as a function of the confidence threshold.

PRECISION-CONFIDENCE

How is it constructed? For all confidence thresholds between 0 and 1:

• compute the confusion matrix for that threshold
• compute the precision associated with this specific matrix
• plot the point on the graph

How should it be interpreted? For example, at a confidence threshold of 0.4, the precision is around 0.92. This indicates that at this threshold, there are relatively few false positives.

Key takeaway: the higher the curve, the better the model.

FURTHER INSIGHTS

This curve must be interpreted carefully because it does not account for false negatives. Moreover, as the threshold increases, fewer predictions are considered, so the increase in precision can be somewhat “artificial”.

This happens because the model becomes very selective. For example, if it only detects one true positive with 98% confidence, it will achieve 100% precision, even though many true positives (below the threshold) are missed.

RECALL-CONFIDENCE

How is it constructed? For all confidence thresholds between 0 and 1:

• compute the confusion matrix for that threshold
• compute the recall associated with this specific matrix
• plot the point on the graph

How should it be interpreted? For example, at a confidence threshold of 0.8, the recall is around 0.78. This indicates that at this threshold, there are many false negatives.

Key takeaway: the higher the curve and the longer it takes to drop, the better the model.

FURTHER INSIGHTS

The curve will inevitably reach 0 because as the confidence threshold increases, the model becomes stricter and accepts fewer predictions. This leads to a sharp increase in false negatives, which drives the recall down.

Precision and recall are both useful metrics but they are complementary. To properly evaluate a model, both must be considered together, especially how one evolves relative to the other.

To avoid constantly switching between the two graphs, combined metrics are used. We will therefore introduce the precision–recall curve and the F1-score in the next section.

The precision-recall curve shows the simultaneous evolution of precision and recall depending on the confidence threshold. It is built in a very specific way, which we will explain just after.

The F1-score is the harmonic mean of precision and recall. This curve is used to find the optimal confidence threshold for deploying your model.

PRECISION-RECALL

How to build it? This curve is constructed point by point because a third dimension is hidden: the confidence threshold.

We start with the leftmost point: this point corresponds to the precision and recall values when the confidence threshold is 1.

Then we plot the next point to the right: we take a threshold of 0.99 and compute precision and recall from the confusion matrix. And so on.

How to read it? For example, consider the point where the confidence threshold is 0.5 (if you stretch the curve between your fingers, this would be the middle). At this threshold, we can read a precision of about 0.65 and a recall of about 0.67.

Key takeaway: the closer the curve gets to the top-right corner, the better the model performs (this example model is actually quite poor, so don’t rely on it).

FURTHER INSIGHTS

The precision-recall curve is used to compute another metric that you will encounter in the results: Average Precision (AP). This metric corresponds to the area under the precision-recall curve. It is the decimal value shown next to each class name in the legend.

By extension, mAP stands for mean Average Precision, which is the average of AP values when multiple classes are involved.

Why is this useful? As mentioned earlier, “getting closer to the top-right corner” is subjective. AP was introduced to quantify how high the curve rises, i.e., how large the area under the curve is. The higher the AP or mAP, the better the model.

EVEN FURTHER INSIGHTS 

After “mAP”, you may see “@0.5”, which corresponds to the threshold value for Intersection over Union (IoU).

To understand IoU, think of it as a verification tool for the model: the model predicts a bounding box, and there is also a ground-truth bounding box. We need a way to quantify how well the predicted box matches the true one.

Consider the two boxes below:

The green box represents ground truth, and the orange box is the model’s prediction. It is not perfectly placed, but it still roughly covers the same object.

To measure this, we compute the ratio between the intersection and the union of the two boxes, as illustrated below:

A threshold must be chosen to accept or reject predictions. For example, with an IoU threshold of 0.5, a predicted box must overlap the ground-truth box by at least 50% to be considered valid.

What is the difference between confidence and IoU?

Step 1: choose thresholds (arbitrarily):

• confidence threshold = 0.5
• IoU threshold = 0.5

Step 2: the model trains, processes images, and produces predictions with associated confidence scores.

Step 3: based on the confidence threshold, the model accepts predictions above the threshold and rejects the others.

Step 4: verification phase. Predicted boxes are compared to ground-truth boxes using IoU:

• If IoU ≥ threshold → true positive
• If IoU < threshold but passed confidence → false positive

Step 5: ground-truth boxes with no matching prediction are counted as false negatives.

The key difference is that the IoU threshold is only used during training, whereas the confidence threshold is crucial during deployment.

The diagram below explains what si going on with an image :

Another metric you may encounter is mAP50-95. It computes mAP across IoU thresholds from 0.5 to 0.95 (step 0.05) and takes the average.

This metric is particularly useful because it is stricter about bounding-box accuracy, making it a standard benchmark in computer vision.

F1-SCORE

The F1-score is the harmonic mean of precision and recall:

One key advantage of F1 is its sensitivity to extreme values. A model with perfect precision but very low recall will still get a poor F1 score.

This curve is used to find the optimal confidence threshold: it corresponds to the x-value of the maximum point on the curve.

Key takeaway: the x-value at the maximum is a good candidate for your confidence threshold.

If the F1 curve forms a plateau instead of a sharp peak:

• Prioritize precision (avoid false positives): choose the right end of the plateau (higher threshold).
• Prioritize recall (avoid false negatives): choose the left end (lower threshold).
• Balanced approach: choose the center of the plateau.

In a multi-class context, the confusion matrix expands to include a row and a column for each category, plus a "background" dimension (corresponfing to the previous negative category) representing the absence of an object. The rows indicate predictions, and the columns represent the ground truth.

The main diagonal retains its significance as correct predictions. Classification errors now occupy all off-diagonal cells. The "background" column lists false detections, while the "background" row identifies real objects that were not detected.

To calculate performance metrics (precision, recall, F1 score) and determine the optimal confidence threshold, it is necessary to decompose this global matrix. The analysis is conducted using a "one-vs-all" approach, a 2x2 binary matrix is "generated" for each class individually, isolating the true positives, false positives and false negatives specific to that category. As previously noted, true negatives are not required for this analysis.

Let us examine how to determine the three metrics, once again using the "Penny" class as an example.

For a given class (e.g. : "penny"), the metrics are defined from the matrix as follows :

• True positives (TP) : the single cell at the intersection of the class's row and column (the diagonal).
• False positives (FP) : the sum of all other cells in the class's row (predicted as the class but belonging to another).
• False negatives (FN) : the sum of all other cells in the class's column (belonging to the class but predicted as another).

These values enable the calculation of precision and recall for a specific confidence threshold. Since each confusion matrix corresponds to a unique threshold, repeating this calculation across all possible thresholds generates the performance curves. This iterative process is repeated for each class, producing individual curves. A summary curve, termed "mean" (represented in bold on the graphs), is then derived from the aggregation of all class-specific curves, offering a global view of the model's performance.

For reference, the optimal confidence threshold value, corresponding to the x-coordinate of the F1 curve's maximum, is directly indicated in the legend, immediately following the "all classes" mention.

NORMALIZED MULTI-CLASS CONFUSION MATRIX

Finally, there is the normalized confusion matrix. It provides an overall view because instead of raw counts, it shows percentages.

How to read it?

The normalized confusion matrix shows detection percentages for each class, so it is read column by column. For example, for “Penny” from top to bottom:

• 1% were predicted as “Dime”
• 5% were confused with “Nickel”
• 82% of “Penny” were correctly recognized
• 12% were not detected at all

The total sums to 100%, so everything is consistent.

labels.jpg is an image containing four plots similar to the one below:

Top left: distribution of the different classes.

Top right: overlay of all bounding boxes.

Bottom left: coordinates of the centers of the bounding boxes.

Bottom right: dimensions of the bounding boxes.

These are statistics about the annotated data, this file is independent of training.

These documents contain the same metrics, one in CSV table form, the other in graphical form.

The result graphs are divided into two groups to evaluate the model's convergence and efficiency:

• the six curves on the left (the losses) : these indicate that the model is learning and making fewer errors.

The metric curves (two columns on the right): Precision, Recall, mAP50, and mAP50-95 illustrate the model's increasing competence. The objective is to observe the progression of these curves until they reach a plateau. Stabilization indicates that the model has reached its maximum potential and that training can cease. Conversely, continuous growth suggests that extending the training is necessary.

• the four curves on the right (precision, recall, mAP50, mAP50-95) : these show that the model is becoming more performant.

The loss curves (three columns on the left): These show the evolution of the error (loss) during training and validation. A simultaneous decrease in both curves is the ideal sign of a model that is learning correctly and generalizing its acquired knowledge effectively.

FOR FURTHER EXPLORATION

Overfitting occurs when the model memorizes the training data by heart instead of learning generalizable patterns. It then becomes incapable of performing well on new data.

• visual detection : this phenomenon is identified by a divergence in the loss curves. If the training curve (top) continues to descend while the validation curve (bottom) rises, the model is overfitting. The validation curve acts as a neutral witness, as the model does not train on these data.
• automatic management and resources : YOLO mitigates this risk by saving two versions: `last.pt` (the final state, potentially overfitted) and `best.pt` (the best weights recorded before validation degradation). Although `best.pt` guarantees performance, overfitting remains an unnecessary consumption of time and computational energy.
• optimization via the 'patience' parameter : to avoid this waste, it is recommended to use the `patience` parameter. This automatically stops training if no improvement is detected on the validation set after a defined number of epochs, ensuring that the process halts as soon as the model reaches its optimum.

The quantity of images generated of this type is proportional to the size of your dataset. These files provide visual feedback on the model's behavior at different stages :

• 'train_batchx.jpg' : illustrates the training samples, i.e., the annotated images ("ground truth") submitted to the model during the learning phase.
• 'val_batchx_labels.jpg' : presents the images from the validation set accompanied by their actual annotations, serving as a reference for evaluation.
• 'val_batchx_pred.jpg' : displays the predictions made by the model on the validation images. The overlay of predicted bounding boxes and identified classes allows for a visual assessment of the model's accuracy (localization and classification) and helps identify potential errors.

If the result files of your model are satisfactory, you can then test your model on other data, especially your “test” folder.

But how do you know if your model is good? Open your results folder and let’s go through the important points to check:

• the mAP50-95 score:
  • where to find it: in the results.csv file. Important point: in this file, the separator is a comma “,” whereas Excel expects semicolons, so the file may appear unreadable. To fix this issue, first open the file in a text editor and add a first line that says “sep=,” as shown in the image below:

• • save the modification, close the file, then open it in Excel.
  • each row corresponds to an epoch and each column contains values for different losses, metrics, and learning rate for that epoch. The column we are interested in is called: metrics/mAP50-95(B).
  • how to use it to evaluate your model: scroll all the way down to get the value of this metric at the last epoch of your model.
  • what is a good value:
    • below 0.3: bad
    • above 0.5: good for complex objects
    • above 0.7: excellent

• loss curves:
  • where to find them: in the results.png file to directly visualize the curves and their trends.
  • what to observe:
    • if both curves decrease and stabilize, your model is good
    • if the validation curves (val) increase while the training curves (train) decrease, your model is bad (sign of overfitting)

• F1 score curve:
  • where to find it: the file BoxF1_curve.png
  • what to check:
    • the curve should be as high as possible, close to 1.0.
  • what to extract:
    • take the x-value of the maximum, as it is the optimal confidence threshold. It is written in the legend next to “all classes” as: “all classes max_value at optimal_threshold”. You take the second value to use as the “conf” parameter when deploying your model.

• valbatchXpred.jpg images:
  • these images correspond to the model output, giving you a visual feedback of training. You can check whether detections are correct, bounding boxes are well placed, and no objects are missed.

VALIDATION ON THE TEST SET

If the model appears satisfactory, evaluate it on unseen data (the "test" folder) :

Loading : import the optimized model ('best.pt') located in 'runs/detect/trainX/weights' :

from ultralytics import YOLO

model = YOLO("path/to/your/file/best.pt") # Do not hesitate to use the absolute path unless you are on the cluster

Configuration : ensure that the `data.yaml` file contains the path to the test set (or create a `data_test.yaml`).

Execution : launch the specific validation :

metrics = model.val(split="test")

Analysis : Consult the results generated in `runs/detect/valX`. If performance is insufficient, retraining is necessary.

RETRAINING AND OPTIMIZATION

To improve an underperforming model, two strategies are possible:

• Extending the Training :
  • You must imperatively load the weights from the last epoch ('last.pt'), not the best ones ('best.pt').
  • Launch the training with the option 'resume=True'. The 'epochs' parameter then indicates the total number of epochs targeted (e.g., to go from 100 to 150 epochs, specify 'epochs=150').

from ultralytics import YOLO

model = YOLO("path/to/your/file/last.pt")

results = model.train(epochs=150, resume=True)

• Adjusting Hyperparameters :

If extending the training proves insufficient, it may be necessary to modify the model's configuration (learning rate, batch size, architecture, etc.). These advanced settings, specific to each use case, are not detailed in this document; it is recommended to consult the official Ultralytics documentation or specialized online resources to explore these aspects further.