Tracking

LUC3D annotates animal pose from several synchronized camera views. Within a single frame, each animal appears as a separate 2D pose instance in each view. Cross-view tracking answers two questions:

1. Correspondence — which 2D instances across the different views belong to the same physical animal? Matching instances are grouped (one per camera) into a cross-view group, which is then triangulated into a single 3D pose.
2. Identity — give each animal a stable identity (id_0, id_1, …) and keep it consistent across frames, so the same animal keeps the same identity and color over time.

The result is per-frame cross-view groups, their 3D reconstructions, and per-frame identity assignments.

The CrossViewTracker

LUC3D's tracker is a faithful port of the CrossViewTracker written by Liezl Maree in the talmolab/sleap-3d repository. It associates each camera's 2D detections to a running list of 3D targets one view at a time, using Hungarian assignment on a cost that blends a 2D reprojection term with a 3D point-to-ray term. It is a greedy assignment tracker — there is no Kalman filter, velocity model, or track aging.

Running tracking

Action	Shortcut	What it does
Track Frame	Shift+T	Tracks only the current frame.
Track All	Cmd+Shift+T / Ctrl+Shift+T	Runs every frame in sequence, carrying targets forward for temporal continuity, and clears existing identities for a fresh run.

Tracking assigns identities, not tracks. To turn identities into per-instance tracks (or vice-versa), use Tracks ▸ Propagate IDs → Tracks / Tracks → IDs after a run — see Tracks & Identities.

Number of animals

On first use you're asked for the animal count:

• A set count caps the tracker at exactly that many 3D targets per frame.
• Blank (auto-detect) uses the largest instance count seen in any single (camera, frame) pair as a cap. This cap keeps the identity pool from drifting upward across frames, while letting the count float below it.

How matching works

For each frame, LUC3D runs the following. At least 2 included cameras are required (see Camera views).

1. Build the detection pool. Collect the instances in each included camera. Optional detection filters can drop instances that are too sparse or low-scoring before matching.
2. Associate one camera at a time. For each camera, build a target×detection cost matrix and solve a single Hungarian assignment. Matched detections are fused into their targets, which are then re-triangulated (DLT) before the next camera is processed. Because each camera refines the 3D targets in turn, later cameras match against progressively better 3D estimates.
3. Birth new targets. Detections that go unmatched but appear in at least 2 views seed new targets, matched to each other by epipolar error.
4. Assign identities. Every target spanning ≥2 views is written out as a cross-view group with an identity. On Track All, targets persist from the previous frame, so identities stay consistent over time without an explicit re-ordering pass.

The association cost

Two terms are summed into the cost that the Hungarian solver minimizes:

• 2D reprojection term — how far each target's current 3D estimate reprojects from the candidate 2D detection, normalized by the velocity threshold and decayed over any frame gap by the time penalty. Weighted by the 2D correspondence weight.
• 3D point-to-ray term — the distance between each target's 3D points and the back-projected ray of the candidate detection, normalized by the distance threshold. Weighted by the 3D correspondence weight.

Both terms honor node weights: each keypoint's contribution is scaled by its node weight, and a node weighted 0 is skipped entirely.

Identity rules

A target spanning fewer than 2 views gets no identity — a lone detection has no cross-view partner and cannot be geometrically verified. Visible instances that landed in no group are marked "no identity" and render in neutral gray, rather than falling back to a stale label.

Triangulation method

Tracking's internal re-triangulation always uses fast DLT. The default triangulation method setting (DLT or Bundle Adjustment) governs the explicit Triangulate / Triangulate All actions, not tracking.

Relationship to Maree et al. (2024)

The Measuring Behavior 2024 paper (Maree et al.) framed multi-view association as exhaustive hypothesis testing — enumerating every possible grouping of instances across views and keeping the one with the lowest reprojection error. That approach is exact but exponential in the number of cameras, and the paper listed a faster, greedy alternative as future work. LUC3D's shipped CrossViewTracker is that greedy alternative: it solves the association with per-view Hungarian assignment instead of enumerating hypotheses, so it stays fast as views and animals scale.

Tuning tracking (Tracking Wizard)

All tracking parameters are edited in the Tracking Wizard and persisted to your browser. They are read at the start of each track run, so a change takes effect on the next Track Frame / Track All.

Open the wizard: Tracks ▸ Tracking Wizard…, Help ▸ Settings ▸ Tracking Wizard, or Cmd+Shift+I / Ctrl+Shift+I. Edits are staged locally and only committed on Apply — Cancel, Esc, or clicking the backdrop discards them.

Node weights

The wizard lists every node in the project skeleton, each with a weight.

Setting	Default	Range	Effect
Per-node weight	1	0–1 (step 0.01)	Scales how much that node contributes to both cost terms (2D reprojection and 3D point-to-ray). 1 = fully considered, 0 = ignored entirely.

Tip

Lower the weight of noisy or hard-to-see nodes (e.g. a tail tip) so they don't pull matches off. Nodes without an explicit override default to 1.

Camera views (view weights)

The wizard's Camera Views section toggles each camera in or out of the tracker.

Setting	Default	Values	Effect
Per-camera weight	1 (included)	0 or 1 (binary)	0 drops the camera from the association math and the 3D solve. Excluded views stay visible in the GUI and are still reprojected into, so you can inspect them.

Warning

View weights are on/off, not a continuous scale. At least 2 views must stay included or tracking cannot run.

Association parameters

Defaults reproduce LUC3D's standard behavior, so leaving them untouched is safe.

Setting	Default	Range	Effect
2D correspondence weight	1	0–20	Weight on the 2D reprojection term of the cost.
3D correspondence weight	6	0–20	Weight on the 3D point-to-ray term. The 3D term is the dominant, most meaningful knob; 6 was the benchmark champion (fewest sustained ID switches).
Velocity threshold	10	0.1–1000	Normalizer for the 2D term (normalized image units). Larger = more tolerant of 2D displacement.
Distance threshold (mm)	50	0.1–1000	Normalizer for the 3D point-to-ray term, in world units (mm). Larger = more tolerant of 3D separation.
Time penalty	0.1	0–10	Exponential decay exp(−timePenalty · Δt) applied to the 2D term over frame gaps. 0 = no time decay.

Distance threshold ≠ reprojection error threshold

The distance threshold is a soft normalizer inside the tracking cost (world-space mm) — it shapes how ray distance becomes cost, and never hard-rejects a match. The reprojection error threshold below is a completely separate, hard pixel outlier filter in the 3D solve. They live in different code paths.

Detection filters

These prune the detection pool before matching. Both default to off.

Setting	Default	Range	Effect
Min visible nodes	0 (off)	0–30	Drop any instance with fewer than N present keypoints. (The reference config used 8 on a 15-node skeleton.)
Min instance score	0 (off)	0–1	When detection scores exist, drop instances scoring below this. (The reference config used 0.85.)

Reprojection error threshold

Setting	Default	Range	Effect
Reprojection error threshold (px)	0 (off)	0–500	Robust-triangulation outlier rejection. After the initial solve, any single node-in-a-view whose reprojection error exceeds this is dropped and the node is re-triangulated from the remaining views; a node left with fewer than 2 reliable views becomes null. Applies to all triangulation, not just tracking. See Triangulation.

Editable keyboard shortcuts

Every tracking action (Track Frame, Track All, Open Tracking Wizard, Triangulate, Find match) is rebindable under Help ▸ Settings ▸ Keyboard Shortcuts. See the shortcut reference.