Winter Term 2020/2021

Lecture: 3D Scanning and Motion Capture

Winter Term 2020/2021
2020, Aug 24    

Lecturer

Justus Thies Angela Dai

Teaching Assistants

Andrei Burov Felix Altenberger

Lecture: 3D Scanning and Motion Capture

Course Website

TUM-Online

Content

3D reconstruction, RGB-D scanning (Kinect, Tango, RealSense), ICP, camera tracking, sensor calibration, VolumetricFusion, Non-Rigid Registration, PoseTracking, Motion Capture, Body-, Face-, and Hand-Tracking, 3D DeepLearning, selected optimization techniques to solve the problem statements (GN, LM, gradient descent).

  • Basic concepts of geometry
    • Meshes (polygonal), Point Clouds, Pixels & Voxels
    • RGB and Depth Cameras
    • Extrinsics and Intrinsics
    • Capture devices
    • RGB and Multi-view
    • RGB-D cameras
    • Stereo
    • Time of Flight (ToF)
    • Structured Light
    • Laser Scanner, Lidar
  • Surface Representations
    • Polygonal meshes (trimeshes, etc.)
    • Parametric surfaces: splines, nurbs
    • Implicit surfaces
    • Ridge-based surfaces
    • Radial basis functions
    • Signed distance functions (volumetric, Curless & Levoy)
    • Indicator function (Poisson Surface Reconstruction)
    • More general: level sets
    • Marching cubes
  • High-level overview of reconstructions
    • Structure from Motion (SfM)
    • Multi-view Stereo (MVS)
    • SLAM
    • Bundle Adjustment
  • Optimization
    • Non-linear least squares
    • Gauss-Newton LM
    • Examples in Ceres
    • Symbolic diff vs auto-diff
    • Auto-diff with dual numbers
  • Rigid Surface Tracking & Reconstruction
    • Pose alignment
    • ICP (point cloud alignment; depth-to-model alignment; rigid ‘fitting’)
    • Online surface reconstruction pipeline: KinectFusion
    • Scalable surface representations: VoxelHashing, OctTrees
    • Loop closures and global optimization
    • Robust optimization
  • Non-rigid Surface Tracking & Reconstruction
    • Surface deformation for modeling
    • Regularizers: ARAP, ED, etc.
    • Non-rigid surface fitting: e.g., non-rigid ICP
    • Non-rigid reconstruction: DynamicFusion/VolumeDeform/KillingFusion
  • Body Tracking & Reconstruction
    • Skeleton Tracking and Inverse Kinematics
    • Learning-based approaches from RGB and RGB-D
  • Face Tracking & Reconstruction
    • Keypoint detection & tracking
    • Parametric / Statistical Models -> BlendShapes
  • Hand Tracking & Reconstruction
    • Parametric Models
    • Some DeepLearning-based things
  • Discriminative vs generative tracking
    • Random forests
  • Motion Capture in Movies
    • Marker-based motion capture
    • LightStage -> movies
  • BRDF and Material Capture
  • Open research questions

Prerequisites

Introduction to Informatics I, Analysis, Linear Algebra, Computer Graphics, C++

# TUM