Spotting a person in one frame is easy, but keeping track of that same person across dozens of frames is the hard part.
Detection tracking is the step that fixes that: it finds objects each frame and then links those sightings into a single, continuous identity.
Think of it like turning a stack of snapshots into a short film, so a camera or car knows who moved where and when.
In this post I’ll explain the simple loop these systems use, which is detect, predict, match, update, and why that loop matters for safety, efficiency, and clearer data.
How Objects Maintain Identity Across Video Frames
Detection tracking is the process of finding objects in each video frame and then following those same objects as they move through later frames. Picture a surveillance camera watching a parking lot, or an autonomous vehicle scanning the road ahead. Each frame gets scanned for objects, and once something is found, the system keeps tabs on it as time passes.
Frame level detection and identity preserving tracking do two different jobs, though people often lump them together. Detection asks “what’s in this frame right now?” while tracking asks “is this the same thing I saw a moment ago?” To answer that second question, the system links detections across frames using motion prediction and data association. It’s basically guessing where an object should be next, then matching it to what actually shows up.
The payoff is simple. Every object gets a persistent ID and a trajectory, a trail showing exactly how it moved through the video from start to finish. Instead of a pile of disconnected boxes, you get a story of movement, one your car, camera, or robot can actually use.
Detection Vs Tracking: Key Differences
Detection looks at one frame at a time and asks what’s there. It draws bounding boxes, labels each object, and attaches a confidence score, all without any memory of what came before. That’s a problem on its own, because a person detected in frame 10 has no built in connection to that same person detected in frame 11. The detector might place them at array index 10 in one frame and index 17 in the next, with nothing tying those two instances together. Left alone, detection loses identity constantly.
Tracking picks up where detection leaves off. It takes those raw detections and threads them into continuous paths, giving each object a stable ID that survives from frame to frame. Single object tracking follows one chosen target through the video, often used when you only care about one thing, like a ball in a sports clip. Multi object tracking handles many objects at once, which is what most real world systems need, whether that’s counting shoppers in a store or watching every car at an intersection.
Here’s what tracking adds on top of detection:
- Speed: tracking reuses prior appearance, location, and velocity, so it’s often faster than rerunning full detection on every single frame.
- Occlusion handling: when a person briefly walks behind a pole, tracking can bridge that gap using motion history, something a lone detector can’t do.
- Identity preservation: tracking keeps the same ID attached to the same object, avoiding the ID switches that plague frame by frame detection.
- Lower computation: many systems run detection every nth frame and let tracking fill in the gaps, cutting workload without losing accuracy.
- Drift correction: periodic redetection resets any small errors that build up while tracking coasts between detection passes.
How The Detection Tracking Pipeline Works Step By Step
Most detection tracking systems follow the same basic loop, frame after frame. It’s a repeating cycle of four moves, each one feeding into the next, and getting familiar with these steps makes the whole approach much less mysterious.
- Detect: The detector scans the current frame and outputs bounding boxes, class labels, and confidence scores for anything it finds. A detection confidence threshold, commonly somewhere between 0.3 and 0.6, filters out weak or unlikely detections before they move forward.
- Predict: Every track already being followed gets a predicted position for the current frame, usually generated by a Kalman filter, a motion prediction tool that’s been around since 1960 and still holds up well for this job.
- Associate: The system now matches new detections to those predicted positions, either through the Hungarian algorithm (a bipartite matching method with roots in the 1950s) or through IoU matching, which compares how much detection boxes overlap with predicted boxes. An NMS IoU threshold around 0.3 to 0.5 also helps clean up overlapping duplicate detections.
- Update and manage: Matched detections update their existing tracks, unmatched detections spin up brand new tracks, and any track that goes unmatched for too long, commonly 3 to 30 consecutive frames, gets deleted.
Here’s how that plays out in practice. Say a detector scans a frame and finds three people, with confidence scores of 0.92, 0.85, and 0.48. Applying a 0.5 threshold drops that third detection, leaving two solid candidates. Hungarian assignment then links those two detections to their matching existing track IDs, based on predicted position and overlap. If one of them doesn’t match anything already being tracked, it starts a fresh track of its own.
Technologies Powering Modern Object Detection And Tracking
CNN based detectors form the backbone of most modern tracking systems. Faster R CNN, introduced in 2015, brought strong accuracy through a two stage detection process. SSD followed in 2016 with a faster, single stage approach. The YOLO family has become the most widely used option, starting in 2015 and evolving through YOLOv3 in 2018, YOLOv4 and YOLOv5 around 2020, YOLOv7 in 2022, and the newer YOLOv8. All of these output the same basic ingredients tracking needs, bounding boxes, class labels, and confidence scores, just with different tradeoffs between speed and accuracy.
Transformer based detection represents a newer direction, with DETR arriving in 2020. Instead of relying on hand crafted components like anchor boxes and non maximum suppression, DETR treats detection as a direct set prediction problem, using attention mechanisms to reason about the whole image at once. It’s a different way of thinking about the same goal, finding objects and describing where they are.
Siamese network trackers take yet another approach, built specifically for single object tracking. SiamFC, SiamMask, and GOTURN all learn what a chosen object looks like early on, then follow that object through later frames using similarity matching rather than running a full detector again and again. This makes them well suited for situations where you’re locked onto one specific target and want to follow it smoothly, without the overhead of detecting everything else in the scene at the same time.
Final Words
You see detection tracking in action when a camera spots a person and keeps the same ID as they move through the scene. This post showed that combo, compared detection and tracking, walked through the Detect-Predict-Associate-Update pipeline, and highlighted modern detectors.
The takeaway: detection finds objects per frame, tracking links detections into persistent IDs, and models like YOLO, DETR, and Siamese trackers make it work in real time.
If you’re asking what is detection tracking, it’s a practical way to turn single-frame detections into clear object trajectories you can test and build on, and it’s ready to try.
FAQ
Q: What is tracking by detection?
A: Tracking by detection means combining frame-level object detection with following those objects over time, linking matching detections across frames—for example, a camera or autonomous vehicle keeping the same person ID as they move.
Q: Can you give me an example of a tracking system? / What is an example of tracking?
A: An example of a tracking system is an autonomous vehicle that detects pedestrians in each frame and links them into persistent tracks, so each person keeps the same ID and path across video.
Q: Is YOLO considered AI?
A: YOLO is considered an AI-based object detector: a deep-learning model that predicts bounding boxes and class scores in single frames and often provides the detection stage used by tracking systems.