---
title: "Object Detection"
description: "Filter real bounding boxes with score thresholds, IoU, and NMS"
source: https://vizlearn.app/ai/vision/object-detection
access: premium (interactive player requires Lifetime; this doc is free)
---

# Object Detection

Object detection predicts a class, a score, and a bounding box for each detected instance in an image. Several objects can share a class while occupying different locations. Score filtering removes weak candidates; non-maximum suppression removes likely duplicates. Raising the score cutoff rejects more candidates, while raising the overlap cutoff tolerates more duplication.

Think of each box as a sticky note on the photograph: “I think there is a cat here.” Two notes can point to the same cat. Removing a redundant note is useful; removing the only note on the other cat is a missed detection. The photograph does not change when you take a note away.

## How Object Detection Works

An image classifier assigns class scores to one supplied image or crop. A detector also predicts **where** each instance is. Its output has a variable number of records. Two records labeled `cat` can be correct when two cats are present; the class name alone cannot tell you whether the records are duplicates.

This example uses one recorded prediction from Torchvision's SSDLite320 detector with a MobileNetV3 backbone and COCO weights. The model receives a resized 320×320 RGB image, computes features, predicts class scores and box offsets, and decodes the offsets into rectangles. Training taught the model the relationship between image features, classes, and locations. Changing a threshold afterward does not train it again. See the [Torchvision model documentation](https://docs.pytorch.org/vision/stable/models/generated/torchvision.models.detection.ssdlite320_mobilenet_v3_large.html) for the model and its pretrained weights.

A box uses two opposite corners, `(x1, y1)` and `(x2, y2)`. Coordinates here are mapped back to the 1280×853 source photo: x increases to the right and y increases downward. The rectangle covers its interior; it does not trace the animal's outline. Predicting an outline belongs to segmentation.

### A score filter removes weak candidates

A candidate passes when its score is **strictly greater than** the score threshold. At 0.50, a score of 0.33 fails and a score of 0.98 passes. The threshold changes eligibility, not the original score or coordinates.

A high score does not prove that a box is correct or unique. Several almost identical cat boxes can all have high scores. The score filter alone leaves those duplicate notes on the photograph.

### IoU measures how much two boxes overlap

Intersection over union, **IoU**, is the area inside both rectangles divided by the area covered by either rectangle:

`IoU = intersection / (area A + area B − intersection)`

The subtraction prevents counting the overlap twice. Disjoint boxes have IoU 0; identical boxes have IoU 1. Two different objects can still have overlapping boxes, especially when they stand close together. IoU measures rectangle overlap, not whether two animals are the same animal.

### Non-maximum suppression keeps an existing box

Greedy **non-maximum suppression**, or NMS, keeps the highest-scoring pending candidate, then suppresses lower-scoring candidates of the **same class** whose IoU with it exceeds the NMS threshold. *Pending* means still awaiting a decision; *suppressed* means removed from the output. NMS repeats with the highest score still pending. The kept box retains its original coordinates; hard NMS does not average boxes or fuse their shapes. This comparison follows the [Torchvision NMS rule](https://docs.pytorch.org/vision/stable/generated/torchvision.ops.nms.html).

The two thresholds have different meanings. Raising the score threshold demands a stronger model score. Raising the IoU threshold tolerates more overlap before a pair qualifies for suppression. If you treat both as “more filtering,” you will turn one in the wrong direction.

## Object Detection Step by Step

The default photograph shows two cats beside each other. The recording includes all **31 foreground candidates with scores above 0.20**, before NMS. That floor bounds this interactive recording; it is not a fundamental limit of object detection.

The default controls are **score threshold 0.50** and **NMS IoU threshold 0.50**. Candidate aliases stay attached to the same predictions throughout the experiment.

| Candidate | Predicted class | Score | Location |
| --- | --- | ---: | --- |
| A | cat | 99.3% | Right cat |
| B | cat | 98.6% | Left cat |
| C | cat | 97.8% | Almost the same right-cat box as A |
| U | bench | 33.1% | A weak model prediction |

First inspect A's record. Its opposite corners are approximately **(639.2, 161.4)** and **(1200.8, 731.5)** in source pixels. Those numbers locate the rectangle. Its score of 99.3% is a separate field, not a measurement of how much of the cat the box covers.

Apply the score filter. **17 of the 31 candidates pass**; U's bench score is too low. A, B, and C all pass, including the redundant right-cat candidate. Scores alone have not solved duplication.

Keep A, the strongest pending candidate. Compare it with C. Their intersection covers roughly **317,207 px²**, and their union covers **325,699 px²**, giving **IoU ≈ 0.9739**. Both predict `cat`, and 0.9739 exceeds 0.50, so suppress C. A's coordinates stay unchanged.

Now compare A with B. Their boxes overlap slightly, with **IoU ≈ 0.0402**. Both predict `cat`, but 0.0402 does not exceed 0.50. B remains pending. It has not yet been selected as the next retained leader.

Applying the same rule to A's remaining candidates suppresses **eight more**. B is now the strongest pending candidate; keep it and suppress **six** overlapping candidates around it. The repetitive comparisons are grouped in the animation, but every removal is computed and retains its cause.

The result is **two detections**, A and B: two records of the same class at different locations. The 15 NMS removals are distinct from the 14 earlier score-filter rejections.

The default playback then runs two controlled experiments. Raise only the score cutoff to **0.99**: B's 98.6% score fails, leaving **one detection** although both cats remain visible. Restore the score cutoff to 0.50, then raise only the IoU cutoff to **0.90**: **three detections** survive, including P, another box around the right cat. The ending restores the default two boxes and keeps both comparisons visible. The model has not changed between these results.

## Choosing Score and IoU Thresholds

Use the default result as a reference, then change one control at a time. Every row below uses the same image, model scores, and box coordinates.

| Score threshold | IoU threshold | Result | Interpretation |
| ---: | ---: | --- | --- |
| 0.50 | 0.50 | A and B | Two separate cat detections |
| 0.99 | 0.50 | A only | B's 98.6% score fails; one visible cat is missed |
| 0.50 | 0.90 | A, B, and P | A lower-score duplicate survives the relaxed overlap cutoff |
| 0.50 | 0.00 | A only | Any positive same-class overlap is enough to suppress; B is lost |
| 0.30 | 0.50 | A, B, and U | The weak bench prediction also survives |
| 1.00 | 0.50 | No detections | No score is strictly above 1.00 |

These are measurements on one model and photograph, not recommended production settings. Choose deployment thresholds using representative labeled images and the cost of missed objects versus false detections. If two nearby objects overlap, aggressively removing overlap can discard a valid detection. If a duplicate's overlap falls below the cutoff, it can survive.

### Diagnose missing boxes before changing a threshold

If an object is missing, first inspect the candidates **before filtering**. A box rejected for low score points to the score cutoff; a box removed because it overlaps a stronger same-class box points to NMS. Lowering the score cutoff cannot recover a candidate that already passes it and is then suppressed. Neither control can produce a box the model never predicted.

For duplicate boxes, inspect their pairwise IoU. A lower IoU cutoff makes that pair easier to suppress, but can also remove a nearby object's box. Try **Too much suppression**: at IoU 0.00, the small A/B overlap is enough to lose B. Compare one setting at a time on labeled images so you can see whether removing duplicates also loses separate objects.

### Postprocessing cost

For n candidates, score filtering costs O(n), sorting costs O(n log n), and straightforward greedy NMS has O(n²) worst-case time complexity. The working candidate lists take O(n) space. These costs describe postprocessing; running the neural network is separate. This player processes at most 31 recorded candidates, while an actual detector can predict many more.

## Edge Cases

- **No detections:** the output is empty, but the photo can still contain objects. Lowering a threshold changes which recorded candidates survive; it cannot create a missing model prediction.
- **Threshold equality:** a score equal to the score threshold fails. IoU equal to the NMS threshold does not cause suppression. The strict comparisons matter at the boundary.
- **Different classes:** class-aware NMS does not suppress one box because it overlaps a retained box of another class. It can therefore retain conflicting class predictions over the same region.
- **Identical boxes:** with the same class and IoU threshold below 1, the lower-scoring identical box is suppressed. At threshold 1, even identical boxes survive this strict rule.
- **Tied scores:** this lesson breaks ties by source ID for deterministic replay. Other implementations or execution devices may resolve tied scores differently.
- **Maximum density:** setting score to 0.20 and IoU to 1.00 retains all 31 recorded candidates. The roster shows at most six rows around the active candidate, or the first six in the final result, and states the visible range and total. The calculation uses every candidate.

## Common Mistakes

- **Counting boxes as objects.** Several retained boxes can describe one cat, while another cat receives none. Compare detections against labeled data before treating a count as reliable.
- **Confusing score with localization quality.** A model can score a class highly while placing an imperfect box. A percentage is not a guarantee that the complete detection is correct.
- **Removing all boxes of one class.** Two cats need two records. Class matching is a condition for this NMS comparison, not sufficient evidence of duplication.
- **Expecting NMS to improve coordinates.** It selects among existing predictions. It does not move A's corners toward the animal or average A and C.
- **Assuming every detector uses this pipeline.** Model families differ in box prediction and duplicate handling. This example demonstrates one SSDLite output with class-aware greedy NMS.

## A Note on Simplification

This is an explainer, not a substitute for formal study of computer vision. It uses one photograph, one pretrained detector, and only the recorded candidates above score 0.20. Training, feature extraction details, anchor regression arithmetic, calibration, tracking, segmentation, and dataset-level evaluation are omitted. The model ran offline; the browser recomputes both thresholds over its unedited predictions. Repetitive overlap comparisons are grouped for pacing. Other images and detector families can behave differently.

The source photograph, [Two cats sitting with legs crossed](https://commons.wikimedia.org/wiki/File:Two_cats_sitting_with_legs_crossed.jpg), is by Irshath 25 and released under CC0. The producer and exported metadata preserve image/model hashes and runtime versions. Continue with [Image Classification](https://vizlearn.app/ai/vision/image-classification) to study the one-crop classification contract, or [Confusion Matrix](https://vizlearn.app/ai/ml-evaluation/confusion-matrix) to separate predictions from ground truth when evaluating a classifier.