3D Video From Capture to Diffusion

by Lucas, Laurent; Loscos, Cé; line; Ré; mion, Yannick

• ISBN13: 9781848215078
• ISBN10: 184821507X
• Edition: 1st
• Format: Hardcover
• Copyright: 2013-11-25
• Publisher: Wiley-ISTE

Summary

While 3D vision has existed for many years, the use of 3D cameras and video-based modeling by the film industry has induced an explosion of interest for 3D acquisition technology, 3D content and 3D displays. As such, 3D video has become one of the new technology trends of this century.

The chapters in this book cover a large spectrum of areas connected to 3D video, which are presented both theoretically and technologically, while taking into account both physiological and perceptual aspects. Stepping away from traditional 3D vision, the authors, all currently involved in these areas, provide the necessary elements for understanding the underlying computer-based science of these technologies. They consider applications and perspectives previously unexplored due to technological limitations.

This book guides the reader through the production process of 3D videos; from acquisition, through data treatment and representation, to 3D diffusion. Several types of camera systems are considered (multiscopic or multiview) which lead to different acquisition, modeling and storage-rendering solutions. The application of these systems is also discussed to illustrate varying performance benefits, making this book suitable for students, academics, and also those involved in the film industry.

Author Biography

Laurent Lucas currently leads the SIC research group and is in charge of the virtual reality platform of the URCA (University of Reims Champagne Ardenne) in France. His research interests include visualization and co-operation between image processing and computer graphics, particularly in 3DTV, and their applications.

Céline Loscos is Professor at the URCA, within the CReSTIC laboratory, and teaches computer science at the University Institute of Technology (IUT) in Champagne Ardenne, France.

Yannick Remion's research interests include dynamic animation, simulation and co-operation between image processing and computer graphics as well as 3D vision.

Table of Contents

Chapter 1. Fundamentals

1.1. Introduction

1.2. A short history

1.2.1. Pinhole model

1.2.2. 3D and binocular vision

1.2.3. Reconstruction

1.3. Stereopsis and 3D physiological aspects

1.4. 3D computer vision

1.5. Conclusion

1.6. Bibliography

Chapter 2. Digital cameras: definitions and principles

2.1. Introduction

2.2. Acquiring light: physics fundamentals

2.2.1. Radiometry and photometry

2.2.1.1. Scene illumination

2.2.2. Wavelengths and color spaces

2.3. Digital cameras

2.3.1. Optical components

2.3.1.1. Camera optics

2.3.1.2. Errors and corrections

2.3.2. Electronic components

2.3.2.1. Camera sensors

2.3.2.2. Digital noise and noise removal algorithms

2.3.3. Main camera functions and control

2.3.3.1. Autobracketing

2.3.4. Image storage formats

2.4. Camera, human vision and color

2.4.1. Adapting optics and electronic to human perception

2.4.2. Color control

2.4.2.1. Camera response

2.4.2.2. Color characterization

2.5. Outperforming

2.5.1. HDR imaging

2.5.2. Hyperspectral acquisition

2.6. Conclusion

2.7. Bibliography

Chapter 3: Multiview acquisition systems

3.1. Introduction: what is a multiview acquisition system?

3.2. Binocular systems

3.3. Lateral or directional multiview systems

3.4. Surrounding or omnidirectional multiview systems

3.5. Hybrid systems: RGBZ and TOF

3.6. Conclusion

3.7. Bibliography

Chapter 4: Shooting and viewing geometry for 3D TV 

4.1. Introduction

4.2. Output geomerty of imaginary relief

4.2.1. Description

4.2.2. Possible modelling

4.3. Capture geometry of imaginary relief

4.3.1. Type of geometry to be used

4.3.2. Possible modelling

4.4. Link between output and capture geometry

4.4.1. Geometric characterization of imaginary relief experience

4.4.2. Distortion models

4.5. Methodology for specifying multiscopic acquisition

4.5.1. Controlling relief distortion

4.5.2. Perfect relief effect

4.6. Implementation in OpenGL

4.7. Conclusion

4.8. Bibliography

Chapter 5: Geometric and colorimetric calibration and rectification

5.1. Introduction

5.2. Camera calibration

5.2.1. Introduction

5.2.2.Camera model

5.2.3. Calibration with a target

5.2.4. Automatic methods

5.3. Radial distortion

5.3.1. Introduction

5.3.2. When should distortion be corrected?

5.3.3. Radiale distortion correction models

5.4. Image rectification

5.4.1. Introduction

5.4.1.1. Problematics

5.4.2. Image-based methods

5.4.3. Camera-based methods

5.4.4. Rectification of more than 2 images simultaneously

5.5. Camera colorimetric aspects

5.5.1. Applyed colorimetry

5.5.2. Camera colorimetric calibration

5.5.2.1. Estimation of F(k) and S(k)

5.5.2.2. In practice

5.6. Conclusion

5.7. Bibliography

Chapter 6: Feature points detection and image matching

6.1. Introduction

6.2. Feature points

6.2.1. Points detectors

6.2.1.1. Differential operators: Autocorrelation, Harris and Hessian

6.2.1.2. Scale invariance using multi-scale analysis

6.2.1.3. Corner intensity model

6.2.2. Contours and feature points detection

6.2.2.1. Shapes detectors

6.2.2.2. Curvature and scale space

6.2.3. Stable regions: IBR, MSER

6.3. Feature point descriptors

6.3.1. Scale-invariant feature transform: SIFT

6.3.2. Gradient Local Orientation Histogram: GLOH

6.3.3. DAISY descriptor

6.3.4. Speeded Up Robust Features: SURF

6.3.5. Multi-scale Oriented PatcheS: MOPS

6.3.6. Shape context

6.4. Image matching

6.4.1. Descriptors matching

6.4.2. Estimation of the geometric transform: matches grouping

6.4.2.1. Generalized Hough Transform

6.4.2.2. Graph matching

6.4.2.3. RANSAC and variants

6.5. Conclusion

Chapter 7: Multi and Stereoscopic matching, depth and disparity

7.1. Introduction

7.2. Difficulties, primitives and density of stereoscopy matching

7.2.1. Difficulties

7.2.2. Primitives and density

7.3. Simplified geometry and disparity

7.4. Description of stereoscopic and multiscopic methods

7.4.1. Algorithms of local and global matching

7.4.2. Principal constraints

7.4.3. Energy costs

7.5. Methods with explicit consideration of occultations

7.5.1. Stereoscopic local method – propagation of seeds

7.5.1.1 Initialization of germs

7.5.1.2 Approach by propagation

7.5.1.3 Regulation by region sounding

7.5.2 Multiscopic global method

7.5.2.1 Formulation of multiscopic matching

7.5.2.2. Energy function and constraint of geometric consistency

7.5.2.3. Global selection and partition construction

7.5.2.4. Results

7.6. Conclusion

7.7. Bibliography

Chapter 8: Multiview reconstruction

8.1. Problematics

8.2. Visual hull-based reconstruction

8.2.1. Methods to extract visual hulls

8.2.2. Reconstruction methods

8.2.3. Improving volume reconstruction

8.2.3.1. Voxel Coloring

8.2.3.2. Space Carving

8.3. Industrial implementation

8.3.1. Hardware acceleration

8.3.2. Results

8.4. Temporal structuration of reconstructions

8.4.1. Extraction of a generic skeleton

8.4.2. Computation of motion fields

8.5. Conclusion

8.6. Bibliography

Chapter 9: Synthesis of intermediate views

9.1. Introduction

9.2. Interpolation/extrapolation view synthesis

9.2.1. Direct and inverse projections

9.2.1.1. Equations of direct projection

9.2.1.2. Direct projection artefacts

9.2.1.3. Inverse projection inverse

9.2.2. Limiting view synthesis artefacts

9.2.2.1. Cracks

9.2.2.2. Ghost outlines

9.2.2.3. Open zones

9.2.3. View interpolation

9.2.3.1. Fusion of virtual views

9.2.3.2. Detection and smoothing of interpolation artefacts

9.2.3.3. Float textures

9.2.3.4. View extrapolation

9.3. Open zone filling

9.3.1. State of the art on 2D inpainting techniques

9.3.1.1. Diffusion-based methods

9.3.1.2. Similarity-based methods

9.3.2. 3D inpainting

9.3.2.1. Crimini et al. [CRI 04] extension to 3D context

9.3.2.2. Global optimisation-based inpainting

9.4. Conclusion

9.5. Bibliography

Chapter 10: Encoding multiview videos

10.1 Introduction

10.2 Compression of stereoscopic videos

10.2.1 3D formats

10.2.1.1 Frame compatible

10.2.1.2 Mixed Resolution Stereo

10.2.1.3 2D-plus-depth

10.2.2 Associated coding techniques

10.2.2.1 Simulcast

10.2.2.2 MPEG-C and H.264/AVC APS

10.2.2.3 H.264/MVC Stereo Profile

10.3 Compression of multiview videos

10.3.1 3D formats

10.3.1.1 MVV and MVD

10.3.1.2 LDI and LDV

10.3.1.3 DES

10.3.2 Associated coding techniques

10.3.2.1 H.264/MVC Multiview Profile

10.3.2.2 LDI-dedicated methods

10.4 Conclusion

10.5 Bibliography

Chapter 11: 3D mesh compression

11.1. Introduction

11.2. Background on coding: The rate-distortion theory

11.3. Multi-resolution coding of surface meshes

11.4. Topological and progressive coding

11.4.1. Mono-resolution compression

11.4.2. Multi-resolution compression

11.4.2.1. Connectivity-driven approaches

11.4.2.2. Geometry-driven approaches

11.5. Mesh Sequences Compression

11.5.1. Definitions

11.5.2. Spatio-temporal prediction methods

11.5.3. Segmentation based methods

11.5.4. Transformation based methods

11.6. Quality assessment: classical and perceptual metrics

11.6.1. Classical metrics

11.6.2. Perceptual metrics

11.7. Conclusion

11.8. Bibliography

Chapter 12: Depth Video Coding Technologies

12.1 Introduction

12.2 Analysis of a depth map characteristics

12.3 Depth video coding tools

12.3.1 Tools that exploit the inherent characteristics of depth maps

12.3.1.1 Above block-level coding tools

12.3.1.2 Block-level coding tools

12.3.2 Tools that exploit the correlation with the associated texture

12.3.2.1 Prediction mode inheritance / selection

12.3.2.2 Prediction information inheritance

12.3.2.3 Spatial transforms

12.3.3 Tools that optimize depth video coding for the virtual views quality

12.3.3.1 View synthesis optimization

12.3.3.2 Distortion models

12.4 Conclusion

12.5 Bibliography

Chapter 13. Stereoscopic watermarking

13.1. Introduction

13.2. Stereoscopic watermarking constraints

13.2.1. Theoretical framework

13.2.2. Properties

13.2.2.1. Transparency

13.2.2.2. Robustness

13.2.2.3. Data payload

13.2.2.4. Computational cost

13.2.3. Corpus

13.2.3.1. Design criteria

13.2.3.2. Processed corpora

13.2.4. Conclusion

13.3. State-of-the-art on stereoscopic watermarking

13.4. Comparative study

13.4.1. Transparency

13.4.1.1. Subjective evaluation

13.4.1.2. Objective evaluation

13.4.2. Robustness

13.4.3. Computational cost

13.4.4. Conclusion

13.5. Conclusion and perspectives

13.6. References

Chapter 14: 3D HD TV and autostereoscopy

14.1.Introduction

14.2.Technological principles

14.2.1.Stereoscopic devices with glasses

14.2.2.Autostereoscopic devices

14.2.3.Optics

14.2.4.Mesurements of autostereoscopic display

14.3. Mixing filters

14.4.Generating and enterlacing views

14.4.1.Virtual view generation

14.4.2.Enterlacing views

14.5. Futur developments

14.6.Conclusion

14.7.Bibliography

Chapter 15:  Augmented and/or mixed reality

15.1. Introduction

15.2. Real-time pose computation

15.2.1. Requirements for pose computation

15.2.2. Model/image feature matching

15.2.2.1. Iterative tracking methods

15.2.2.2. Recognition methods

15.2.2.3. Real-time constraint

15.2.3. Pose computation: the main PnP algorithms

15.2.3.1. Reprojection error minimization

15.2.3.2. Direct methods

15.2.4. Pose computation and planar surfaces

15.3. Model acquisition

15.3.1. Offline modelization

15.3.2. Online modelization

15.4. Conclusion

15.5. Bibliography

Chapter 16. Visual comfort and visual fatigue for stereoscopic restitution

16.1. Introduction

16.2. Visual comfort and fatigue: definition and evaluation

16.2.1. Visual fatigue

16.2.2. Visual comfort and discomfort

16.2.3. Assessment and evaluation of fatigue and discomfort

16.3. Symptoms and signs of fatigue and discomfort

16.3.1. Ocular and oculomotor fatigue

16.3.2. Cognitive fatigue

16.3.3. Symptoms and signs of discomfort

16.4. Sources of fatigue and discomfort

16.4.1. Ocular constraints

16.4.2. Cognitive constraints

16.5. Application to 3D displays and contents

16.5.1. Comfort zone

16.5.2. Restitution defects

16.5.3. Accommodation and blur

16.5.4. Visual attention

16.5.5. Null or erroneous motion parallax

16.5.6. Exposure duration and training effects

16.6. Predicting visual fatigue and discomfort: emerging models

16.7. Conclusion

16.8. Bibliography/references

Chapter 17: 2D to 3D conversion

17.1.Introduction

17.2. 2D-3D conversion workflow

17.3.Content preparation for conversion

17.3.1.Depth script

17.3.2. Video advantage on fix images

17.3.3. Automatic conversion decoy

17.3.4.Special cases of automatic conversion

17.3.5.Optimal content for 2D-3D conversion

17.4. Conversion steps

17.4.1. Segmentation step

17.4.2.Depth map, computation and propagation

17.4.3.Missing image generation

17.5.3D-3D conversion

17.6.Conclusion

17.7.Bibliography

Chapter 18. 3D-model retrieval

18.1. Introduction

18.2. General principles of shape retrieval

18.3. Global 3D-shape descriptors

18.3.1. Shape descriptor histogram

18.3.2. Spherical harmonics

18.4. 2D-view based methods

18.5. Local 3D-shape descriptors

18.5.1. 3D-shape spectrum descriptor

18.5.2. 3D-shape context

18.5.3. Spin-images

18.5.4. Heat cernel Signature

18.6. 3D-shape similarities

18.6.1. Reeb graphs

18.6.2. Bof-of-Words

18.7. 3D-shape retrieval in 3D-videos

18.7.1. Action recognition in 3D-videos

18.7.2. Facial expression recognition in 3D-videos

18.8. Performance evaluation of shape retrieval methods

18.8.1. Statistic tools for evaluation

18.8.2. Benchmarks

18.9. Applications

18.9.1. Browsing in a collection of 3D-models

18.9.2. Modeling by example

18.9.3. Decision aid

18.9.4. 3D-face recognition

18.10. Conclusion

18.11. References

Chapter 19: 3D HDR images and videos: acquisition and restitution

19.1. Introduction

19.2. HDR and 3D acquisition

19.2.1. Subspace 1D: HDR images

19.2.2. Subspace 2D: HDR videos

19.2.3. Subspace 2D: 3DHDR images

19.2.3.1. Stereo matching for HDR reconstruction

19.2.3.2. Discussion on color data consistency

19.2.4. Extension to the whole space: 3DHDR videos

19.3. 3D HDR rendering

19.3.1. Rendering on a 3D-dedicated display

19.3.2. Rendering on an HDR-dedicated display

19.4. Conclusion

19.5. Bibliography

Chapter 20: 3D TV visualization for life sciences

20.1.Introduction

20.2.Scientific visualization

20.2.1. 3D construction

20.2.2.Interactivity

20.2.3.3D visualization

20.3.Medical imaging

20.3.1.Volumic visualization

20.3.2.3D medical imaging

20.3.2.1.Teaching

20.3.2.2.Diagnostic

20.3.2.3.Therapy

20.4. Molecular modeling

20.4.1.Classical modes of visualization

20.4.2.Molecular modeling in relief

20.5.Conclusion

20.6.Bibliography

Chapter 21: 3D reconstruction of sport scenes

21.1.Introduction

21.2.Automatic selection of region of interest

21.2.1.Region of interest role and caracteristics

21.2.2.Color space segmentation

21.2.3.Spacial consistency

21.3.Primitive extraction by HOUGH transform

21.3.1.Ellipsoid segment detection

21.4.Primitive/model matching

21.4.1.Line beams

21.5.Conclusion

21.6.Bibliography

Chapter 22: Experimental, live retransmissions in stereoscopic 3D (S-3D)

22.1.Introduction

22.2.Show retransmissions

22.3.Surgery retransmissions

22.4. Steadicam magazine retransmissions

22.5.Transatlantic video-presentation retransmission

22.6.Bicycle competition retransmissions

22.7.Conclusion

22.8.Bibliography