Tracking¶
LUC3D annotates animal pose from several synchronized camera views. Within a single frame, each animal appears as a separate 2D pose instance in each view. Cross-view tracking answers two questions:
- Correspondence — which 2D instances across the different views belong to the same physical animal? Matching instances are grouped (one per camera) into a cross-view group, which is then triangulated into a single 3D pose.
- Identity — give each animal a stable identity (
id_0,id_1, …) and keep it consistent across frames, so the same animal keeps the same identity and color over time.
The result is per-frame cross-view groups, their 3D reconstructions, and per-frame identity assignments.
The CrossViewTracker
LUC3D's tracker is adapted from the CrossViewTracker written by Liezl Maree in the talmolab/sleap-3d repository. It associates each camera's 2D detections to a running list of 3D targets using Hungarian assignment on a cost that blends a 2D reprojection term with a 3D point-to-ray term. It is a greedy assignment tracker — there is no Kalman filter, velocity model, or track aging. LUC3D adds two things to the reference implementation: stale-detection eviction and frame-synchronous association (both described below).
Running tracking¶
| Action | Where | What it does |
|---|---|---|
| Track Frame | Toolbar button, Shift+T | Tracks only the current frame (no history from other frames). Replaces that frame's groups; existing identities are kept. |
| Track Frame Range… | Track Frame ▾ dropdown | Opens a dialog with a Start / End frame slider and an Animals field. Tracking restarts at the first frame of the range; identities and groups are cleared only inside the range, and existing identity names are reused so re-runs don't mint new ones. Use it to re-track a stretch under different Tracking Wizard settings. |
| Track All | Toolbar button, Ctrl+Shift+T | Clears every identity and group, then runs every frame in sequence, carrying targets forward for temporal continuity. |
Tracking assigns identities, not tracks. To turn identities into per-instance tracks (or vice-versa), use Tracks ▸ Propagate IDs → Tracks / Propagate Tracks → IDs after a run — see Tracks & Identities. Triangulate and Triangulate All create groups from identities automatically, so you can triangulate straight after tracking.
Number of animals¶
On the first Track Frame / Track All you're asked for the animal count:
- A set count caps the tracker at exactly that many 3D targets per frame.
- Blank (auto-detect) uses the largest instance count seen in any single (camera, frame) pair as a cap. This keeps the identity pool from drifting upward across frames, while letting the count float below it. On a lazily-loaded project this is sampled from the frames currently in memory.
To change the count later, use the Animals field in the Track Frame Range dialog (blank resets to auto-detect).
How matching works¶
For each frame, LUC3D runs the following. At least 2 included cameras are required (see Camera views).
- Evict stale detections. Each target remembers its last detection per camera. Any per-camera detection older than the stale detection eviction window is dropped, so a target that was occluded in one view doesn't keep matching against an out-of-date 2D anchor.
- Build the detection pool. Collect the instances in each included camera. Optional detection filters can drop instances that are too sparse or low-scoring before matching.
- Associate camera by camera. For each camera, build a target×detection cost matrix and solve a single Hungarian assignment. Every camera in the frame is scored against the same 3D state from the start of the frame; matched targets are re-fused (DLT) once at the end of the frame.
- Birth new targets. Detections that go unmatched in at least 2 views seed new targets, matched to each other by epipolar error.
- Assign identities. Every target spanning ≥2 views is written out as a cross-view group with an identity. On Track All, targets persist from the previous frame, so identities stay consistent over time without an explicit re-ordering pass.
The association cost¶
Two terms are summed into the cost that the Hungarian solver minimizes:
- 2D reprojection term — how far each target's current 3D estimate reprojects from the candidate 2D detection, normalized by the velocity threshold and decayed over any frame gap by the time penalty. Weighted by the 2D correspondence weight.
- 3D point-to-ray term — the distance between each target's 3D points and the back-projected ray of the candidate detection, normalized by the distance threshold. Weighted by the 3D correspondence weight.
Both terms honor node weights: each keypoint's contribution is scaled by its node weight, and a node weighted 0 is skipped entirely.
Identity rules
A target spanning fewer than 2 views gets no identity — a lone detection has no cross-view partner and cannot be geometrically verified. Visible instances that landed in no group are marked "no identity" and render in neutral gray, rather than falling back to a stale label.
Triangulation method
Tracking's internal re-triangulation always uses fast DLT. The default triangulation method setting (DLT or Refined) governs the explicit triangulate actions (this frame and all frames), not tracking.
Relationship to Maree et al. (2024)¶
The Measuring Behavior 2024 paper (Maree et al.) framed multi-view association as exhaustive hypothesis testing — enumerating every possible grouping of instances across views and keeping the one with the lowest reprojection error. That approach is exact but exponential in the number of cameras, and the paper listed a faster, greedy alternative as future work. LUC3D's shipped CrossViewTracker is that greedy alternative: it solves the association with per-view Hungarian assignment instead of enumerating hypotheses, so it stays fast as views and animals scale.
Tuning tracking (Tracking Wizard)¶
All tracking parameters are edited in the Tracking Wizard and persisted to your browser. They are read at the start of each track run, so a change takes effect on the next Track Frame / Track Frame Range / Track All.
Open the wizard: Tracks ▸ Tracking Wizard…, Help ▸ Settings ▸ Tracking Wizard, or Ctrl+Shift+I. It has three collapsible sections — Node Weights, Camera Views, and Tracking Thresholds. Edits are staged and only committed on Apply — Cancel, Esc, or clicking the backdrop discards them.
Node Weights¶
The wizard lists every node in the project skeleton, each with a weight.
| Setting | Default | Range | Effect |
|---|---|---|---|
| Per-node weight | 1 |
0–1 (step 0.01) |
Scales how much that node contributes to both cost terms (2D reprojection and 3D point-to-ray). 1 = fully considered, 0 = ignored entirely. |
Tip
Lower the weight of noisy or hard-to-see nodes (e.g. a tail tip) so they don't pull matches off.
Camera Views (view weights)¶
The Camera Views section toggles each camera in or out of the tracker.
| Setting | Default | Values | Effect |
|---|---|---|---|
| Per-camera weight | 1 (included) |
0 or 1 |
0 drops the camera from the association math and the 3D solve. Excluded views stay visible in the GUI and are still reprojected into, so you can inspect them. |
Warning
View weights are on/off, not a continuous scale. At least 2 views must stay included or tracking cannot run.
Tracking Thresholds¶
Defaults reproduce LUC3D's standard behavior, so leaving them untouched is safe.
| Setting | Default | Range | Effect |
|---|---|---|---|
| Min visible nodes (filter) | 0 (off) |
0–30 |
Drop any instance with fewer than N present keypoints before matching. |
| Min instance score (filter) | 0 (off) |
0–1 |
When detection scores exist, drop instances scoring below this before matching. |
| 2D correspondence weight | 1 |
0–20 |
Weight on the 2D reprojection term of the cost. |
| 3D correspondence weight | 6 |
0–20 |
Weight on the 3D point-to-ray term. The 3D term is the dominant, most meaningful knob; 6 gave the fewest sustained ID switches in benchmarks. |
| Velocity threshold | 10 |
0.1–1000 |
Normalizer for the 2D term (normalized image units). Larger = more tolerant of 2D displacement. |
| Distance threshold (mm) | 25 |
0.1–1000 |
Normalizer for the 3D point-to-ray term, in world units. Larger = more tolerant of 3D separation. |
| Time penalty | 0.1 |
0–10 |
Exponential decay exp(−timePenalty · Δt) applied to the 2D term over frame gaps. 0 = no time decay. |
| Stale detection eviction (frames) | 20 |
0–500 |
Drop a target's per-camera detection once it is older than this many frames, so a target doesn't anchor to a view where it has been occluded. 0 = never evict (reference behavior). |
| Reprojection error threshold (px) | 0 (off) |
0–500 |
Robust-triangulation outlier rejection. After the initial solve, any single node-in-a-view whose reprojection error exceeds this is dropped and the node is re-triangulated from the remaining views; a node left with fewer than 2 reliable views becomes null. Applies to all triangulation, not just tracking. See Triangulation. |
Distance threshold ≠ reprojection error threshold
The distance threshold is a soft normalizer inside the tracking cost (world-space mm) — it shapes how ray distance becomes cost, and never hard-rejects a match. The reprojection error threshold is a completely separate, hard pixel outlier filter in the 3D solve.
Stale eviction and identity swaps
Together, the stale window (20 frames) and the tighter distance threshold (25 mm) cut sustained identity swaps after occlusions by roughly 3–5× on the benchmark datasets used to tune them. If your animals move slowly or occlusions are long, try a larger window.
Editable keyboard shortcuts
Track Frame, Track All, Open Tracking Wizard, and Triangulate are rebindable under Help ▸ Settings ▸ Keyboard Shortcuts. See the shortcut reference.