Skip to content

Triangulation

LUC3D is a multi-view pose-annotation tool. In each frame, the same subject is labeled in several calibrated, fixed camera views. Triangulation fuses those 2D labels into a single 3D estimate for every keypoint, then projects that 3D point back into each camera so you can see how well the reconstruction agrees with the original labels (the reprojection).

LUC3D offers two methods:

Method Speed Accuracy Minimum views
DLT (Direct Linear Transform) Instant Good 2
Refined (Ref) — bundle adjustment ~1.2× DLT About the same as DLT; better when one view's 2D point is clearly wrong 2

You set a default in Help ▸ Settings ▸ Triangulation Wizard, which drives every implicit triangulation (the T shortcut, Edit ▸ Triangulate Current Frame, and automatic re-triangulation after an edit). The toolbar Triangulate ▾ menu's Method choice (DLT or Ref) sets the same default. The Instances tab shows which method produced the current result.

Refined is bundle adjustment

Refined is bundle adjustment with the cameras held fixed: their calibration is never adjusted, only the 3D points are refined. Earlier builds labeled it Bundle Adjustment (BA). Saved projects record the method per group, so old files load unchanged.

DLT and Refined side by side

Both methods triangulate each node of the skeleton on its own (the sketches show one node, from above) and start the same way: each camera's 2D prediction is a line of sight, and the projection, the 3D point, is where those lines meet. Real detections never meet exactly. Here camera 3's 2D prediction is badly wrong: an outlier, marked stray. The other three are off too, by less, as real predictions always are. The green cross is the target: the true position. The reprojection is where the projection lands back in a camera's 2D image, and the reprojection error is the distance on that 2D image between the prediction and the reprojection. The bars in each sketch add up those errors: Refined's sum is lower than DLT's, even though its error in the stray camera is larger. The dark red line is the ground truth error, how far the projection is from the target, which shrinks from DLT to Refined. A sketch can show it because it knows the target; on a real recording only the reprojection error can be measured.

Direct Linear Transformation (DLT), per node
one direct solve · fast

DLT sketch: four cameras seen from above. The projection is pulled toward the stray camera's line of sight and lands off in every camera.

Solves one set of linear equations (algebraic least squares) for the point that fits every line at once, treating every camera alike. Fast, but an outlier pulls the projection, so every camera ends up off.

Refined (Bundle adjustment), per node
slower

Refined sketch: the projection starts at the DLT point and moves closer to the target, with the stray camera still counted a little.

Bundle adjustment: starts at the DLT point, projects it back into every camera and moves it to shrink the reprojection errors. A camera that disagrees a lot counts less, which helps when one view's 2D prediction is clearly wrong: here the projection lands closer to the target than DLT's, though the outlier, still counted a little, keeps it off. The reprojection error threshold removes the outlier altogether. When the views roughly agree, it gives about the same point as DLT, sometimes slightly worse. The cameras stay fixed; only the point moves. Slower.


Per node: a camera is counted less only for the node where its 2D point disagrees. If one camera misplaces the nose, it still counts for every other node it sees well.

╌╌ line of sight: grey if the camera is a little off, orange if stray  ·  ─ line from the camera to the projection

The camera layout is made up, but the projections and errors in these sketches are what LUC3D's own solvers give for it. The same figure is in Help ▸ Settings ▸ Triangulation Wizard, where clicking a sketch picks the method. That panel's bottom bar always shows the method and reprojection error threshold Triangulate All will use, and a value you changed but haven't applied turns blue and shows what it was.

DLT — Direct Linear Transform

DLT recovers a 3D point by solving a single linear system built from the camera projections and the 2D observations. It minimizes an algebraic error in homogeneous coordinates rather than the geometric error that actually matters visually.

This makes DLT fast, unique, and usable with as few as two views. The trade-off: even with perfect labels, the recovered point can reproject a fraction of a pixel to a couple of pixels away from the labels. DLT is effectively instantaneous and is a fine default while labeling.

Refined

Refined is bundle adjustment with the cameras held fixed. It starts from the DLT solution and minimizes the geometric reprojection error — the actual pixel distance between each 2D label and where the 3D point lands when projected back into that camera. It helps most when one view's 2D prediction is clearly wrong (see DLT and Refined side by side). When the views roughly agree it lands about where DLT does, sometimes slightly further from the true position: its reported error is never larger than DLT's, but that error is measured on the same 2D points it fits, so a lower number does not always mean a better 3D point (see How the two phases weigh the cameras). It is slower than DLT.

Per-point optimization

A key simplification in LUC3D: the cameras are fully calibrated and fixed. Their intrinsics, pose (extrinsics), and lens distortion are known and never adjusted. Because the cameras don't move, every 3D point can be optimized independently — there is no shared state coupling one keypoint's solution to another's.

So Refined reduces to a per-point nonlinear least-squares refinement: for a single point it adjusts only its three world coordinates (X, Y, Z), minimizing the reprojection residuals across every view in which the point is visible.

Initialization

  1. DLT first. The point is initialized with the DLT solution — close, and free.
  2. Supplied estimate. If a finite prior 3D position is available (e.g. re-triangulating an edited group), Refined can start there instead.
  3. Failure. If fewer than two valid observations exist, or the initial estimate is degenerate (e.g., resolves to infinity), triangulation reports failure for that keypoint.

The optimization algorithm

LUC3D uses Levenberg–Marquardt (LM), the standard workhorse for nonlinear least-squares. LM blends two behaviors with a single damping parameter: cautious gradient descent when far from the answer, and fast Gauss–Newton steps when close.

Each iteration:

  1. Build a local model of how reprojection error changes as the 3D point moves. The sensitivities (including the lens-distortion terms) are computed analytically, not by finite differences.
  2. Propose a damped step and check whether it lowers the total error.
  3. Accept or retry. If the error drops, the step is accepted and damping is relaxed. If not, the step is rejected and damping is increased — up to 8 retries per iteration.

The solve runs in two phases, then a guard:

Phase Objective Purpose
1. Robust Soft-L1 (pseudo-Huber) loss, scale 15 px A badly mislabeled view is down-weighted instead of dragging the point
2. Polish Plain sum of per-view pixel errors — the metric the app reports Seeded from the better of DLT and phase 1
Guard Compare against the DLT seed Backtracks toward DLT (up to 8 halvings) and falls back to DLT if nothing improves, so Refined's reported error is never larger than DLT's (that error is measured on the points it fits, not against the true position)

How the two phases weigh the cameras

A toy example seen from above: one node and seven cameras around it. Click a camera, or untick it in the table, to leave it out of the solve. Drag a camera's slider to misplace its 2D point, or start from one of the buttons. Each card shows how far that method's answer is from the true position, and the table shows how much each method lets every camera count.

  • DLT gives every camera the same say, so one camera that is far off pulls the answer with it.
  • Phase 1 (robust) counts a camera less the further off it is: about 0.7 at 15 px, 0.35 at 40 px. While every camera has only small errors, it lands where DLT does.
  • Phase 2 (polish) weighs each camera by 1 / its error, so it leans on the cameras it already fits best. A camera that is far off then barely moves the answer, and the reported error always drops. With only small errors, though, polish tends to settle where two lines of sight cross and fits their noise: with Everyday errors it reports the lowest error yet lands furthest from the true position.

About this figure

The app always runs both phases; "robust phase only" is shown for comparison. The figure is a simplified 2D re-implementation that runs in your browser, not LUC3D's solver.

Tuning and stopping criteria

Parameter Value Role
Maximum iterations 20 Upper bound on LM passes per point per phase
Convergence tolerance 1e-8 Threshold on step size and relative error change
Initial damping 1e-3 Mildly Gauss–Newton-biased start
Damping decrease (on success) × 0.3 (floored at 1e-12) Lets accepted steps grow bolder
Damping increase (on failure) × 10 Makes rejected retries smaller/safer
Damping ceiling 1e12 If exceeded, the point is treated as converged/stuck
Retry attempts per iteration 8 Damping adjustments before giving up the iteration
Robust loss scale 15 px Soft-L1 transition in phase 1

LM stops as soon as any of these holds: the step becomes smaller than the tolerance; the relative error improvement falls below the tolerance; no improving step is found; the damping hits its ceiling; or the iteration cap is reached.

Camera and projection model

LUC3D uses a standard pinhole-plus-distortion camera model:

  • Intrinsics — focal lengths and principal point map 3D rays to pixels.
  • Extrinsics — a rotation and translation place each camera in the world. Rotations may be supplied as compact Rodrigues vectors or full rotation matrices.
  • Distortion — a five-coefficient radial-and-tangential lens model (the OpenCV convention).

Distortion handling

Raw 2D labels on disk are stored in distorted (real-lens) pixel coordinates and are never modified.

  • DLT works in undistorted space: each label is undistorted first (an iterative inversion of the lens model, run for 10 passes) so the linear system is clean.
  • Refined projects the 3D estimate through the full distortion model and measures residuals directly against the raw labels in native pixel space — the same error the app reports. The Instances tab also shows the equivalent undistorted-space error as "undist".
  • When the 3D point is projected back for display, the projection is re-distorted so the on-screen markers line up with the original raw labels.

Partial visibility and robustness

  • Missing views. A keypoint not labeled in a camera simply contributes no residual there; only views with a valid observation participate.
  • Minimum views. A point needs at least two valid observations, or it is skipped.
  • User-excluded keypoints. Right-click a node to mark it not visible in that view. Nulled observations are left out of triangulation without altering the underlying data.
  • Outliers. DLT weights every view equally, so one badly mislabeled view pulls the 3D estimate. Refined's soft-L1 loss counts such a view less, for that node only, which helps when one view is clearly wrong: often the answer then fits the views that agree. It does not remove the view, though, and what is left of its pull can still keep the answer off, as in the sketch above. To drop gross outliers automatically, enable the robust-triangulation threshold below.

Robust triangulation

An optional reprojection error threshold adds automatic outlier rejection on top of either solver. It is off by default (0). Set it in the toolbar's Triangulate ▾ menu, or in the Tracking Wizard ▸ Tracking Thresholds ("Reprojection error threshold (px)"). The figure below is also in Help ▸ Settings ▸ Triangulation Wizard, where trying a value only changes the sketch and its Use button sets the threshold.

When the threshold is greater than 0, after the initial solve LUC3D, for each node:

  1. Computes the node's reprojection error in every view it was observed in: the distance on that view's 2D image between the prediction and the reprojection.
  2. If any view is over the threshold and the node still has more than 2 views, drops the single worst one and re-triangulates the node from the rest. Only the worst goes, because each re-solve moves the 3D point and changes every other view's error: a view that was over can land within afterwards.
  3. Repeats until nothing more is dropped.
  4. A node whose remaining views still exceed the threshold gets no 3D point in that frame, rather than one trusted to a bad fit.

The figure shows one node and the same four cameras as the sketches above, seen from above, with one sketch per solve. Every solve uses the same method, and a camera that has been dropped is drawn faded. The threshold is the band on each camera's image plane: the reprojection must land within it, that many pixels either side of the prediction (green within, grey over, orange dropped). Drag the threshold, which runs from off through 640 px down to 1 px on a log scale, or type a value, and switch the first solve between DLT and Refined, to see which cameras are kept. As above, the layout is made up and the numbers are what LUC3D's own solvers give for it.

  • DLT is pulled away by the stray, so it misses by a lot. Moving the slider right from off, nothing happens until 53 px, where camera 3 is dropped: the other cameras are all within that. At 15 px another camera is over too, but only the worst goes. Either way, solved again without it, the other three land near the target but still a little wrong, because they are a little off too.
  • Refined already lands closer, with the stray still counted a little. Dropping the stray brings it closer still, but not onto the target. Without the threshold, or at 69 px or more here, the stray keeps counting.
  • 100 to 15 px is the recommended range. Past it you can go too low: a threshold (here, under 9 px for DLT and under 14 px for Refined) also drops a camera that was only a little off, so the projection rests on just two cameras. Lower still (under 6 px for Refined), the last two still miss, so the node has no 3D point in that frame: more gaps in the tracks. (Seen from above, two lines of sight cross; in 3D they just miss.)

This works per node within a view: a dropped camera still counts for every other node it sees well, and a whole camera view is never dropped (that is the Camera views job). It applies to every triangulation: This frame, All frames, a frame range, and the automatic re-triangulation after an edit. It does not change how Track All matches animals across cameras: the tracker's own solves always use plain DLT on every view.

Numerical safeguards

  • Behind-the-camera check — projections from behind a camera (vanishing depth) are treated as invalid rather than producing a meaningless pixel.
  • Singular-system check — a numerically singular LM solve is rejected; LM responds by increasing damping and retrying.
  • Point-at-infinity check — DLT initialization detects and rejects solutions that resolve to infinity.
  • Monotone guard — Refined never returns a larger reported reprojection error than DLT.

How triangulation fits into the workflow

Triangulation operates on groups — sets of instances across cameras representing the same subject (see Tracking). If a frame has identities but no groups (e.g. right after Track All), Triangulate builds the groups from the identities first. It is invoked in several ways:

  • Current frame — triangulate every group in the visible frame (T, Edit ▸ Triangulate Current Frame, or This frame in the toolbar Triangulate ▾ menu).
  • All frames — clicking the toolbar Triangulate ▾ button (or All frames in its menu) sweeps every frame with groups, yielding to the interface every 100 frames so the app stays responsive, then reports the average reprojection error in the status bar.
  • Frame range — Edit ▸ Triangulate Multiple Frames does the same for a chosen start/end and camera subset.
  • Automatic re-triangulation — when you move or null a keypoint, the affected group is re-triangulated immediately using whichever method it last used. If re-triangulation fails, the previous reprojections are restored.

After triangulation, LUC3D stores the 3D points and their per-view reprojections, updates the 3D viewport, and surfaces the mean reprojection error (in pixels) so you can judge reconstruction quality at a glance.

Practical guidance

  • DLT by default, Ref for clearly wrong views. DLT is a fine default. Switch to Ref when one view's 2D prediction may be clearly wrong; when the views roughly agree, Ref gives about the same 3D points (sometimes slightly worse) for ~20% more time. A lower reported error from Ref is not by itself proof of a better 3D point.
  • Watch the reprojection error. It's the best signal of reconstruction quality.
  • Null bad observations. A high error on one node usually points to a single mislabeled view — right-click it, or enable the reprojection error threshold to drop such views automatically.
  • More views help. Two is the minimum; additional views improve both robustness and accuracy.