Triangulation

LUC3D is a multi-view pose-annotation tool. In each frame, the same subject is labeled in several calibrated, fixed camera views. Triangulation fuses those 2D labels into a single 3D estimate for every keypoint, then projects that 3D point back into each camera so you can see how well the reconstruction agrees with the original labels (the reprojection).

LUC3D offers two methods:

Method	Speed	Accuracy	Minimum views
DLT (Direct Linear Transform)	Instant	Good	2
Bundle Adjustment (BA)	Slower	Best	2

You set a default in Settings ▸ Default Triangulation, which drives every implicit triangulation (the keyboard shortcut, the Edit menu, and automatic re-triangulation after an edit). The toolbar dropdowns also let you pick a method explicitly per action.

DLT — Direct Linear Transform

DLT recovers a 3D point by solving a single linear system built from the camera projections and the 2D observations. It minimizes an algebraic error in homogeneous coordinates rather than the geometric error that actually matters visually.

This makes DLT fast, unique, and usable with as few as two views. The trade-off: even with perfect labels, the recovered point can reproject a fraction of a pixel to a couple of pixels away from the labels.

DLT is effectively instantaneous and is a fine default while labeling.

Bundle Adjustment

Bundle Adjustment refines the result by minimizing the geometric reprojection error — the actual pixel distance between each 2D label and where the 3D point lands when projected back into that camera. This yields a more accurate reconstruction, especially when a point is observed across many views or when label noise is uneven.

Per-point optimization

A key simplification in LUC3D: the cameras are fully calibrated and fixed. Their intrinsics, pose (extrinsics), and lens distortion are known and never adjusted. Because the cameras don't move, every 3D point can be optimized independently — there is no shared state coupling one keypoint's solution to another's.

So while the technique is called "bundle adjustment," in LUC3D it reduces to a per-point nonlinear least-squares refinement. For a single point, BA adjusts only its three world coordinates (X, Y, Z), minimizing the sum of squared reprojection residuals across every view in which the point is visible.

Initialization

BA always starts from a good guess so it converges quickly:

1. DLT first. By default, the point is initialized with the DLT solution — close, and free.
2. Supplied estimate. If a finite prior 3D position is available, BA can start there instead.
3. Failure. If fewer than two valid observations exist, or the initial estimate is degenerate (e.g., resolves to infinity), triangulation reports failure for that keypoint.

The optimization algorithm

LUC3D uses Levenberg–Marquardt (LM), the standard workhorse for nonlinear least-squares. LM blends two behaviors with a single damping parameter: cautious gradient descent when far from the answer, and fast Gauss–Newton steps when close.

Each iteration:

1. Build a local model of how reprojection error changes as the 3D point moves. The required sensitivities are computed analytically, not by finite differences, keeping them stable and noise-free.
2. Propose a damped step and check whether it lowers the total reprojection error.
3. Accept or retry. If the error drops, the step is accepted and damping is relaxed (bolder next step). If not, the step is rejected and damping is increased (smaller, safer) — LM retries up to 8 times per iteration.

This repeats until convergence or the iteration cap. In typical multi-view setups, BA converges in only a handful of iterations.

Tuning and stopping criteria

Parameter	Value	Role
Maximum iterations	20	Upper bound on LM passes per point
Convergence tolerance	1e-8	Threshold on step size and relative error change
Initial damping	1e-3	Mildly Gauss–Newton-biased start
Damping decrease (on success)	× 0.3 (floored at 1e-12)	Lets accepted steps grow bolder
Damping increase (on failure)	× 10	Makes rejected retries smaller/safer
Damping ceiling	1e12	If exceeded, the point is treated as converged/stuck
Retry attempts per iteration	8	Damping adjustments before giving up the iteration

BA stops as soon as any of these holds: the step becomes smaller than the tolerance; the relative error improvement falls below the tolerance; no improving step is found; the damping hits its ceiling; or the iteration cap is reached.

Camera and projection model

LUC3D uses a standard pinhole-plus-distortion camera model:

• Intrinsics — focal lengths and principal point map 3D rays to pixels.
• Extrinsics — a rotation and translation place each camera in the world. Rotations may be supplied as compact angle-axis vectors or full rotation matrices.
• Distortion — a five-coefficient radial-and-tangential lens model (the OpenCV convention).

Distortion handling

Raw 2D labels on disk are stored in distorted (real-lens) pixel coordinates and are never modified. Triangulation works in undistorted space so the geometry is clean:

• Before triangulating, each observed label is undistorted (an iterative inversion of the lens model, run for 10 passes).
• BA minimizes reprojection error in undistorted pixel space.
• When the 3D point is projected back for display, the projection is re-distorted so the on-screen markers line up with the original raw labels.

Partial visibility and robustness

• Missing views. A keypoint not labeled in a camera simply contributes no residual there; only views with a valid observation participate.
• Minimum views. A point needs at least two valid observations, or it is skipped.
• User-excluded keypoints. A keypoint can be marked "nulled" in a specific view (e.g., to suppress a mislabeled detection). Nulled observations are left out of triangulation without altering the underlying data.
• No robust loss in the solve itself. Within a single DLT/BA solve, all residuals are weighted equally — there is no Huber/Tukey down-weighting. A single badly mislabeled 2D point can pull the 3D estimate, so the workflow relies on you reviewing reprojection error, nulling bad observations, or enabling the robust-triangulation filter below.

Robust triangulation

An optional reprojection error threshold adds automatic outlier rejection on top of the solve. It is off by default (0) and set in the Tracking Wizard ("Reprojection error threshold (px)").

When the threshold is greater than 0, after the initial solve LUC3D iteratively:

1. Computes each node's reprojection error in every view it was observed in.
2. Drops the single worst over-threshold observation per node and re-triangulates that node from the remaining views. (Removing one at a time is necessary because each exclusion shifts every other view's error.)
3. Repeats until no node has an over-threshold observation, never dropping a node below 2 views.
4. Any node still left with fewer than 2 reliable views triangulates to null — it is dropped from 3D rather than trusted to a bad fit.

This works per node within a view; it never drops a whole camera view (that is the Camera views / view weights job). Because it operates on the triangulation itself, it affects all triangulation — explicit Triangulate actions and the tracker's internal solves alike.

Numerical safeguards

• Behind-the-camera check — projections from behind a camera (vanishing depth) are treated as invalid rather than producing a meaningless pixel.
• Singular-system check — a numerically singular LM solve is rejected; LM responds by increasing damping and retrying.
• Point-at-infinity check — DLT initialization detects and rejects solutions that resolve to infinity.
• Analytic sensitivities — closed-form derivatives avoid finite-difference noise.

How triangulation fits into the workflow

Triangulation operates on groups — sets of instances across cameras representing the same subject (see Tracking). BA or DLT is invoked in several ways:

• Current frame — triangulate every group in the visible frame.
• All frames — sweep every frame with groups, yielding to the interface periodically (~every 100 frames) so the app stays responsive, then reporting an average reprojection error.
• Frame range — the same, bounded to a start/end frame.
• Automatic re-triangulation — when you move or null a keypoint, the affected group is re-triangulated immediately using whichever method it last used. If re-triangulation fails, the previous reprojections are restored.

After triangulation, LUC3D stores the 3D points and their per-view reprojections, updates the 3D viewport, and surfaces the mean reprojection error (in pixels) so you can judge reconstruction quality at a glance.

Practical guidance

• Use DLT for speed, BA for accuracy. DLT is a fine default while labeling; switch to BA when you need the tightest reprojection error or before exporting final 3D data.
• Watch the reprojection error. It's the best signal of reconstruction quality. BA should generally beat DLT on the same labels.
• Null bad observations. Because there is no automatic outlier rejection, a high reprojection error usually points to a single mislabeled view.
• More views help. Two is the minimum; additional views improve both robustness and accuracy.