depth3d-lab: depth, 3D reconstruction and view synthesis on nuScenes-mini

Charts

Track A: AbsRel, raw metric output vs per-image median scaling

Track A: AbsRel per camera (raw metric output)

Track A: AbsRel per camera (median-scaled = shape error)

Track A: AbsRel per depth bin (raw / scaled)

Track A: median scale factor GT/pred per camera (1.0 = metric scale correct)

Track C: held-out PSNR (higher is better)

Track C: held-out SSIM (higher is better)

Track C: held-out LPIPS alex (lower is better)

Track C: training loss and number of Gaussians

Track B: AbsRel on identical reference pixels, mean with 95% bootstrap CI

Track B: VGGT relative pose errors vs nuScenes GT (per view pair)

Track E: PSNR vs real next frame, 95% CI over held-out pairs

Track E: LPIPS (alex) vs real next frame, 95% CI over held-out pairs

Track E: training loss

Track D: SHARP render vs real frame, PSNR and LPIPS

Depth comparison viewer

Colour scale: log depth 1-80 m (turbo, red = near). AbsRel on this frame's LiDAR pixels is shown on the right.

3D point cloud: VGGT vs LiDAR

VGGT point map of CAM_FRONT in scene-0061 (114,040 points, coloured by the image, scaled to LiDAR with one sequence scale) and the LiDAR return of the same keyframe (11,609 points, white). Drag to orbit, wheel to zoom. Toggle: VGGT LiDAR

Trained 3D Gaussian Splatting scene

491,520 of 736,227 Gaussians from the 'lidar_depth0.2' variant (15.7 MB .splat). Drag to orbit, right-drag to pan, wheel to zoom. Moving cars and pedestrians appear as ghosts because the scene is modelled as static.

Renders

3DGS held-out previews

gs_lidar_15k_cam_back_left.jpg (left: held-out photo, right: render)

gs_lidar_15k_cam_front.jpg (left: held-out photo, right: render)

gs_lidar_cam_back_left.jpg (left: held-out photo, right: render)

gs_lidar_cam_front.jpg (left: held-out photo, right: render)

gs_lidar_depth0.05_15k_cam_back_left.jpg (left: held-out photo, right: render)

gs_lidar_depth0.05_15k_cam_front.jpg (left: held-out photo, right: render)

gs_lidar_depth0.05_cam_back_left.jpg (left: held-out photo, right: render)

gs_lidar_depth0.05_cam_front.jpg (left: held-out photo, right: render)

gs_lidar_depth0.2_15k_cam_back_left.jpg (left: held-out photo, right: render)

gs_lidar_depth0.2_15k_cam_front.jpg (left: held-out photo, right: render)

gs_lidar_depth0.2_cam_back_left.jpg (left: held-out photo, right: render)

gs_lidar_depth0.2_cam_front.jpg (left: held-out photo, right: render)

gs_random_15k_cam_back_left.jpg (left: held-out photo, right: render)

gs_random_15k_cam_front.jpg (left: held-out photo, right: render)

gs_random_cam_back_left.jpg (left: held-out photo, right: render)

gs_random_cam_front.jpg (left: held-out photo, right: render)

3DGS camera path

Track E: real vs generated next frames

Track D: SHARP renders

sharp_scene-0061_next_keyframe.jpg

sharp_scene-0655_next_keyframe.jpg

All result tables (docs/RESULTS.md)

Results

Generated by python scripts/make_results.py from results/*.json. Do not edit by hand. All timings measured on an NVIDIA L4 (23 GB). Dataset: nuScenes v1.0-mini (10 scenes, 404 keyframes).

Track A - monocular metric depth vs LiDAR (zero-shot)

LiDAR ground truth: 2,424 images (404 keyframes x 6 cameras), points with 1 m < z < 80 m after z-buffering. Per camera (points per image, median depth):

Overall (all LiDAR points, all cameras)

'raw' is the metric output as predicted; 'median-scaled' multiplies each image by median(GT)/median(pred) so that the remaining error is shape error only. 'depth_pro_gtfocal' feeds the true nuScenes focal length instead of the model's own focal estimate. Timing is the median model time per 1600x900 image.

Scale factor and focal estimate per camera

A scale factor of 1.0 means the metric output is already on the LiDAR scale for that camera.

Per camera group (front / side / back) - AbsRel and d<1.25

Per camera - AbsRel raw / scaled

Per depth bin - AbsRel and RMSE

Reading the scale error (generated from the tables above)

depth_pro: per-image median scaling lowers AbsRel from 0.257 to 0.178, i.e. 31% of the raw error is a per-image scale error and the rest is shape error. Worst camera group raw: side (AbsRel 0.270), best: front (0.191). Median scale factor CAM_FRONT 1.12 vs CAM_BACK 0.94. Focal estimate / true focal: CAM_FRONT 1.05, CAM_BACK 1.28. Fraction of LiDAR pixels where the raw prediction exceeded 80 m (clipped before scoring): 0.08%.

depth_pro_gtfocal: per-image median scaling lowers AbsRel from 0.277 to 0.178, i.e. 36% of the raw error is a per-image scale error and the rest is shape error. Worst camera group raw: side (AbsRel 0.306), best: front (0.202). Median scale factor CAM_FRONT 1.16 vs CAM_BACK 1.17. Focal estimate / true focal: CAM_FRONT 1.05, CAM_BACK 1.28. Fraction of LiDAR pixels where the raw prediction exceeded 80 m (clipped before scoring): 0.06%.

dav2_metric_outdoor_large: per-image median scaling lowers AbsRel from 0.582 to 0.194, i.e. 67% of the raw error is a per-image scale error and the rest is shape error. Worst camera group raw: back (AbsRel 1.034), best: front (0.399). Median scale factor CAM_FRONT 0.73 vs CAM_BACK 0.47. Fraction of LiDAR pixels where the raw prediction exceeded 80 m (clipped before scoring): 0.00%.

Per-image AbsRel, mean with 95% bootstrap CI over images

Track C - 3D Gaussian Splatting on one driving scene (gsplat, own training loop)

Scene choice: moving annotations (box velocity > 0.5 m/s) per scene; selected scene-0916.

Scene scene-0916, 216 train / 30 held-out views (every 8th timestamp (offset 4), all 6 cameras), images 800x450 (downscale 2), 7000 steps, seed 0, NVIDIA L4.

LiDAR init vs random init: 2.89 dB PSNR on held-out views (positive favours LiDAR init).

lidar_depth0.05 vs lidar: 1.26 dB PSNR, -0.054 LPIPS (negative LPIPS favours depth supervision).

lidar_depth0.2 vs lidar: 2.22 dB PSNR, -0.083 LPIPS (negative LPIPS favours depth supervision).

Held-out PSNR per camera

Track B - feed-forward multi-view reconstruction (VGGT) vs LiDAR

Model: VGGT-1B (facebook/VGGT-1B), bf16 autocast, input 518x294. Mono reference: Depth Pro on the same reference image and LiDAR pixels. GPU: NVIDIA L4.

Setting (i) 6-camera rig, one timestamp, reference CAM_FRONT

40 sequences, 120,040 LiDAR points on the reference views, 1333 ms per sequence (NVIDIA L4, incl. Depth Pro and metrics).

Paired difference on identical pixels, VGGT (one scale per sequence) minus Depth Pro (per-image scale): -0.001 [-0.024, 0.017]; VGGT with per-image scale minus Depth Pro: -0.012 [-0.029, 0.003]. Negative means VGGT is better. Fraction of sequences where VGGT (seq scale) is lower: 0.42.

Relative camera pose vs nuScenes GT over 200 view pairs: rotation error mean 8.12 deg (median 4.65), translation direction error mean 48.6 deg (median 35.9, over the 200 pairs whose GT baseline exceeds 0.1 m; 0 pairs with a stationary ego vehicle have no defined direction and are excluded); mean GT baseline 0.81 m. Median sequence scale factor (LiDAR / VGGT units): 23.42.

Setting (ii) 5 consecutive CAM_FRONT keyframes, reference = centre frame

40 sequences, 120,495 LiDAR points on the reference views, 1192 ms per sequence (NVIDIA L4, incl. Depth Pro and metrics).

Paired difference on identical pixels, VGGT (one scale per sequence) minus Depth Pro (per-image scale): -0.010 [-0.026, 0.007]; VGGT with per-image scale minus Depth Pro: -0.005 [-0.020, 0.012]. Negative means VGGT is better. Fraction of sequences where VGGT (seq scale) is lower: 0.70.

Relative camera pose vs nuScenes GT over 160 view pairs: rotation error mean 0.31 deg (median 0.14), translation direction error mean 2.3 deg (median 2.0, over the 120 pairs whose GT baseline exceeds 0.1 m; 40 pairs with a stationary ego vehicle have no defined direction and are excluded); mean GT baseline 4.24 m. Median sequence scale factor (LiDAR / VGGT units): 29.68.

Track E - next-frame synthesis with conditional flow matching (plain PyTorch)

Data: 2,342 CAM_FRONT frames (keyframes + sweeps, 404 keyframes) at 128x64; 1,865 train / 467 test consecutive pairs (median dt 0.10 s); held-out scenes scene-0655, scene-1094.

Training fm: 12.4M-parameter UNet, 2,150 steps, batch 64, lr 0.0002, 40.0 min on NVIDIA L4 (world size 1; the two trainings shared the GPU), final logged loss 0.0430.

Training regression: 12.4M-parameter UNet, 2,150 steps, batch 64, lr 0.0002, 40.0 min on NVIDIA L4 (world size 1; the two trainings shared the GPU), final logged loss 0.0025.

Evaluation on 467 held-out pairs, sampler Euler, 50 steps, 173.8 ms per sampled frame (NVIDIA L4).

Paired LPIPS, flow-matching sample minus regression UNet: -0.022 [-0.025, -0.019] (negative = sample is perceptually closer).

Paired LPIPS, sample minus copy-current-frame: 0.059 [0.057, 0.061]; paired PSNR: -0.56 [-0.73, -0.40] dB.

FID of unconditional samples vs held-out frames: 390.3 (467 generated, 467 real). unconditional samples (condition zeroed) vs held-out real frames, 128x64 upsampled to 299 by torchmetrics; small n, treat as indicative only

Track D - single-image 3D Gaussians (Apple SHARP) rendered at the true next pose

10 CAM_FRONT keyframes (one per scene); Gaussians predicted from frame t, rendered at the identity pose and at the true relative pose of the next frame (nuScenes ego pose + calibration). Metrics only on pixels the render covers. GPU: NVIDIA L4.

Prediction time 5496 ms per image, render 22.0 ms per view (NVIDIA L4).