Tutorial: 3D Reconstruction¶
This tutorial covers the complete workflow from 2D annotations to 3D pose reconstruction, including grouping, triangulation, validation, and export.
Prerequisites¶
Before starting 3D reconstruction, you need:
- [x] Multi-camera videos loaded
- [x] Camera calibration loaded (TOML or JSON)
- [x] 2D annotations in at least 2 camera views per subject
- [x] Skeleton loaded with correct node/edge definitions
Step 1: Group Instances Across Views¶
Triangulation requires knowing which instances across cameras represent the same individual. You do this by creating Instance Groups.
Manual Grouping¶
- Click an unlinked instance in Camera 1 to enter assignment mode — it gets highlighted
- Click the corresponding instance (same individual) in Camera 2
- Click in Camera 3, Camera 4, etc. as needed
- Press Shift+G (or click Group in the toast) to create the group
To unlink a group, select it and press Shift+U.
The instances are now linked. You'll see them listed together in the Instances tab.
Auto-Grouping by Track¶
If you've assigned tracks consistently across views:
- Go to Edit → Group by Track & Triangulate All
- Choose the tracks and cameras to include and click Triangulate
- LUC3D groups instances that share the same track across cameras and triangulates every frame
Auto-Grouping by Identity¶
Similarly, if you've assigned identities:
- Go to Edit → Group by Identity & Triangulate All
- Groups instances by identity and triangulates
After tracking
If you ran the cross-view tracker, skip this step — Triangulate builds groups from identities automatically.
Step 2: Triangulate¶
Single Frame¶
With groups created, press T to triangulate the current frame using the default method from Help ▸ Settings ▸ Triangulation Wizard (DLT unless you changed it). Hover the toolbar Triangulate ▾ button for This frame, All frames and the Method choice (DLT or Ref), which sets the same default.
For every group in the frame, LUC3D:
- Gathers each keypoint's 2D positions from all cameras in the group
- Solves for the 3D position (DLT: fast linear solve; Ref: refines it to minimize reprojection error)
- Stores the 3D points on the Instance Group
- Computes reprojected 2D positions for all cameras
Batch Triangulation¶
- Edit → Triangulate Multiple Frames — Specify a frame range and cameras
- Triangulate ▾ (toolbar) — Click it (or pick All frames in its menu) to triangulate every frame that has groups or identities
- Edit → Group by Track & Triangulate All — Groups by track and triangulates every frame
Step 3: Validate with Reprojection¶
After triangulation, check quality using reprojection error.
Enable Visualizations¶
- Press R to show reprojected instances (dashed outlines with white nodes on each camera view)
- Press E to draw error vectors from each labeled node to its reprojection
Read the Error¶
In the Instances tab, the Reprojection Error section shows the frame mean, per-camera means, and a per-instance node breakdown (node × camera table), all color-coded:
- Green (< 2 px): Excellent consistency
- Amber (2–5 px): Acceptable
- Red (≥ 5 px): Needs attention
Investigate High Error¶
If a keypoint has high error:
- Check the instance's node × camera table — which camera is the outlier?
- Go to that camera view and compare:
- Your annotated keypoint position (solid node)
- The reprojected position (dashed overlay)
- If the reprojection looks correct, adjust your annotation toward it
- Re-triangulate (T) to update
Step 4: Inspect in 3D¶
Press \ to open the 3D viewport.
What to Check¶
- Skeleton shape — Does the 3D skeleton look anatomically correct?
- Scale — Is the skeleton the right size relative to the camera positions?
- Limb lengths — Are bone lengths consistent and reasonable?
- Joint angles — No impossible bends or self-intersections?
Navigation¶
- Orbit: Left-click + drag
- Zoom: Scroll wheel
- Pan: Right-click + drag
- Snap to camera: Click a camera frustum to select it, then Show Camera View to see the scene from that camera's perspective
- Reset view: Click Show Initial View, or View → Reset 3D View / Fit 3D to Scene
Environment Overlay¶
If your session folder has environment annotations — .slp/.h5 files with .externals. in the name in each camera folder, e.g. arena corners — click Set Env to triangulate them and show the result as a persistent overlay for scale and position reference. Click Clear Env to remove it.
Step 5: Iterate¶
The reconstruction workflow is iterative:
Repeat until reprojection errors are consistently low (< 2–3 px). If one view's 2D points are clearly wrong, switch the Method in the toolbar Triangulate ▾ menu to Ref.
Common Fixes¶
| Problem | Solution |
|---|---|
| High error on one keypoint | Reposition it in the camera with highest error |
| High error in one camera | Review all keypoints in that camera view |
| High error everywhere | Check grouping — wrong instances may be linked |
| 3D skeleton looks wrong | Compare against reprojection, fix the worst outliers |
| Inconsistent bone lengths | Check for left/right swaps or identity swaps |
Step 6: Export 3D Data¶
Once satisfied with the reconstruction:
Export 3D Points¶
File → Export 3D Points (H5) — Saves triangulated XYZ coordinates as HDF5:
/points_3d — Shape: (n_frames, n_tracks, n_nodes, 3)
/reprojection_error — Shape: (n_frames, n_tracks, n_nodes)
/frame_indices, /track_names, /node_names
See Exporting Data for details.
Export Reprojections¶
File → Export Reprojections (H5) — Saves the 2D reprojected positions for analysis.
Export 2D Labels¶
File → Export SLEAP File Per Session — Exports corrected 2D annotations as one SLEAP file per camera for the active session. Useful for retraining SLEAP models. File → Export SLEAP File By Cam pools several sessions into one file per camera. Run Tracks → Propagate IDs → Tracks first so tracker identities are exported as tracks.
Export a Video¶
File → Export Video Overlays renders the camera views with poses and reprojections burned in; File → Export 3D Video renders the 3D viewport.
Tips for Best Results¶
More cameras = better accuracy
Keypoints visible in 3+ cameras produce more robust triangulations than 2-camera reconstructions.
Mark hidden keypoints
Right-click to mark keypoints not visible rather than guessing positions. Guessed positions add noise to triangulation.
Check calibration quality
If all keypoints consistently have high error, the issue may be with calibration rather than annotations. Try recalibrating.
Use the 3D viewport often
The 3D view catches errors that aren't obvious in 2D — like a limb bending the wrong way or passing through the body.