Skip to content

Tutorial: 3D Reconstruction

This tutorial covers the complete workflow from 2D annotations to 3D pose reconstruction, including grouping, triangulation, validation, and export.

Prerequisites

Before starting 3D reconstruction, you need:

  • [x] Multi-camera videos loaded
  • [x] Camera calibration loaded (TOML or JSON)
  • [x] 2D annotations in at least 2 camera views per subject
  • [x] Skeleton loaded with correct node/edge definitions

Step 1: Group Instances Across Views

Triangulation requires knowing which instances across cameras represent the same individual. You do this by creating Instance Groups.

Manual Grouping

  1. Click an unlinked instance in Camera 1 to enter assignment mode — it gets highlighted
  2. Click the corresponding instance (same individual) in Camera 2
  3. Click in Camera 3, Camera 4, etc. as needed
  4. Press Shift+G (or click Group in the toast) to create the group

To unlink a group, select it and press Shift+U.

The instances are now linked. You'll see them listed together in the Instances tab.

Auto-Grouping by Track

If you've assigned tracks consistently across views:

  1. Go to Edit → Group by Track & Triangulate All
  2. Choose the tracks and cameras to include and click Triangulate
  3. LUC3D groups instances that share the same track across cameras and triangulates every frame

Auto-Grouping by Identity

Similarly, if you've assigned identities:

  1. Go to Edit → Group by Identity & Triangulate All
  2. Groups instances by identity and triangulates

After tracking

If you ran the cross-view tracker, skip this step — Triangulate builds groups from identities automatically.

Step 2: Triangulate

Single Frame

With groups created, press T to triangulate the current frame using the default method from Help ▸ Settings ▸ Triangulation Wizard (DLT unless you changed it). Hover the toolbar Triangulate ▾ button for This frame, All frames and the Method choice (DLT or Ref), which sets the same default.

For every group in the frame, LUC3D:

  1. Gathers each keypoint's 2D positions from all cameras in the group
  2. Solves for the 3D position (DLT: fast linear solve; Ref: refines it to minimize reprojection error)
  3. Stores the 3D points on the Instance Group
  4. Computes reprojected 2D positions for all cameras

Batch Triangulation

  • Edit → Triangulate Multiple Frames — Specify a frame range and cameras
  • Triangulate ▾ (toolbar) — Click it (or pick All frames in its menu) to triangulate every frame that has groups or identities
  • Edit → Group by Track & Triangulate All — Groups by track and triangulates every frame

Step 3: Validate with Reprojection

After triangulation, check quality using reprojection error.

Enable Visualizations

  • Press R to show reprojected instances (dashed outlines with white nodes on each camera view)
  • Press E to draw error vectors from each labeled node to its reprojection

Read the Error

In the Instances tab, the Reprojection Error section shows the frame mean, per-camera means, and a per-instance node breakdown (node × camera table), all color-coded:

  • Green (< 2 px): Excellent consistency
  • Amber (2–5 px): Acceptable
  • Red (≥ 5 px): Needs attention

Investigate High Error

If a keypoint has high error:

  1. Check the instance's node × camera table — which camera is the outlier?
  2. Go to that camera view and compare:
    • Your annotated keypoint position (solid node)
    • The reprojected position (dashed overlay)
  3. If the reprojection looks correct, adjust your annotation toward it
  4. Re-triangulate (T) to update

Step 4: Inspect in 3D

Press \ to open the 3D viewport.

What to Check

  • Skeleton shape — Does the 3D skeleton look anatomically correct?
  • Scale — Is the skeleton the right size relative to the camera positions?
  • Limb lengths — Are bone lengths consistent and reasonable?
  • Joint angles — No impossible bends or self-intersections?
  • Orbit: Left-click + drag
  • Zoom: Scroll wheel
  • Pan: Right-click + drag
  • Snap to camera: Click a camera frustum to select it, then Show Camera View to see the scene from that camera's perspective
  • Reset view: Click Show Initial View, or View → Reset 3D View / Fit 3D to Scene

Environment Overlay

If your session folder has environment annotations — .slp/.h5 files with .externals. in the name in each camera folder, e.g. arena corners — click Set Env to triangulate them and show the result as a persistent overlay for scale and position reference. Click Clear Env to remove it.

Step 5: Iterate

The reconstruction workflow is iterative:

Annotate → Group → Triangulate → Check Error → Fix Annotations → Re-triangulate

Repeat until reprojection errors are consistently low (< 2–3 px). If one view's 2D points are clearly wrong, switch the Method in the toolbar Triangulate ▾ menu to Ref.

Common Fixes

Problem Solution
High error on one keypoint Reposition it in the camera with highest error
High error in one camera Review all keypoints in that camera view
High error everywhere Check grouping — wrong instances may be linked
3D skeleton looks wrong Compare against reprojection, fix the worst outliers
Inconsistent bone lengths Check for left/right swaps or identity swaps

Step 6: Export 3D Data

Once satisfied with the reconstruction:

Export 3D Points

File → Export 3D Points (H5) — Saves triangulated XYZ coordinates as HDF5:

/points_3d           — Shape: (n_frames, n_tracks, n_nodes, 3)
/reprojection_error  — Shape: (n_frames, n_tracks, n_nodes)
/frame_indices, /track_names, /node_names

See Exporting Data for details.

Export Reprojections

File → Export Reprojections (H5) — Saves the 2D reprojected positions for analysis.

Export 2D Labels

File → Export SLEAP File Per Session — Exports corrected 2D annotations as one SLEAP file per camera for the active session. Useful for retraining SLEAP models. File → Export SLEAP File By Cam pools several sessions into one file per camera. Run Tracks → Propagate IDs → Tracks first so tracker identities are exported as tracks.

Export a Video

File → Export Video Overlays renders the camera views with poses and reprojections burned in; File → Export 3D Video renders the 3D viewport.

Tips for Best Results

More cameras = better accuracy

Keypoints visible in 3+ cameras produce more robust triangulations than 2-camera reconstructions.

Mark hidden keypoints

Right-click to mark keypoints not visible rather than guessing positions. Guessed positions add noise to triangulation.

Check calibration quality

If all keypoints consistently have high error, the issue may be with calibration rather than annotations. Try recalibrating.

Use the 3D viewport often

The 3D view catches errors that aren't obvious in 2D — like a limb bending the wrong way or passing through the body.