projects people publications

Facial Expression Acquisition, Tracking and Learning

We address fundamental issues regarding the use of high quality dense 3-D data samples undergoing motions at video speeds, e.g. human facial expressions. In order to utilize such data for motion analysis and re-targeting, correspondences must be established between data in different frames of the same faces as well as between different faces. We investigate data driven approaches that consists of:

The accuracy and resolution of our method allows us to capture and track subtle expression details.

3D Surface Matching and Recognition Using Conformal Geometry

[publications & media]

3D surface matching is a fundamental issue in computer vision with many applications such as shape registration, 3D object recognition and classification. However, surface matching with noise, occlusion and clutter is a challenging problem. In this paper, we analyze a family of conformal geometric maps including harmonic maps, conformal maps and least squares conformal maps with regards to 3D surface matching. As a result, we propose a novel and computationally efficient surface matching framework by using least squares conformal maps. According to conformal geometry theory, each 3D surface with disk topology can be mapped to a 2D domain through a global optimization and the resulting map is a diffeomorphism, i.e., one-to-one and onto. This allows us to simplify the 3D surface-matching problem to a 2D image-matching problem, by comparing the resulting 2D conformal geometric maps, which are stable, insensitive to resolution changes and robust to occlusion and noise. Therefore, highly accurate and efficient 3D surface matching algorithms can be achieved by using conformal geometric maps. Finally, the performance of conformal geometric maps is evaluated and analyzed comprehensively in 3D surface matching with occlusion, noise and resolution variation. We also provide a series of experiments on real 3D face data that achieve high recognition rates.

Publications
Media

High Resolution Tracking of Non-Rigid 3D Motion of Densely Sampled Data Using Harmonic Maps

[publications & media]

We present a novel fully automatic method for high resolution, non-rigid dense 3D point tracking. High quality dense point clouds of non-rigid geometry moving at video speeds are acquired using a phase-shifting structured light ranging technique. To use such data for the temporal study of subtle motions such as those seen in facial expressions, an efficient non-rigid 3D motion tracking algorithm is needed to establish inter-frame correspondences. The novelty of this paper is the development of an algorithmic framework for 3D tracking that unifies tracking of intensity and geometric features, using harmonic maps with added feature correspondence constraints. While the previous uses of harmonic maps provided only global alignment, the proposed introduction of interior feature constraints guarantees that non-rigid deformations will be accurately tracked as well. The harmonic map between two topological disks is a diffeomorphism with minimal stretching energy and bounded angle distortion. The map is stable, insensitive to resolution changes and is robust to noise. Due to the strong implicit and explicit smoothness constraints imposed by the algorithm and the high-resolution data, the resulting registration/ deformation field is smooth, continuous and gives dense one-to-one inter-frame correspondences. Our method is validated through a series of experiments demonstrating its accuracy and efficiency.

Publications
Media

High Resolution Acquisition, Learning and Transfer of Dynamic 3-D Facial Expressions

[publications & media]

Synthesis and re-targeting of facial expressions is central to facial animation and often involves significant manual work in order to achieve realistic expressions, due to the difficulty of capturing high quality dynamic expression data. We address fundamental issues regarding the use of high quality dense 3-D data samples undergoing motions at video speeds, e.g. human facial expressions. In order to utilize such data for motion analysis and re-targeting, correspondences must be established between data in different frames of the same faces as well as between different faces. We present a data driven approach that consists of four parts:

The accuracy and resolution of our method allows us to capture and track subtle expression details. The low dimensional representation of motion data in a unified embedding for all the subjects in the database allows for learning the most discriminating characteristics of each individual's expressions as that person's "expression style" Thus new expressions can be synthesized, either as dynamic morphing between individuals, or as expression transfer from a source face to a target face, as demonstrated in a series of experiments.

Publications
Media

Image-Driven Re-targeting and Relighting of Facial Expressions

[publications]

Synthesis and re-targeting of facial expressions is central to facial animation and often involves significant manual work in order to achieve realistic expressions, due to the difficulty of capturing high quality expression data. Recent progress in dynamic 3-D scanning allows very accurate acquisition of dense point clouds of facial geometry and texture moving at video speeds. Often the new facial expressions need to be rendered in different environments where the illumination is different from the original capture conditions. In this paper we examine the problem of re-targeting captured facial motion under different illumination conditions when the information we have about the face we want to animate is minimal, a single input image. Given an input image of a face, a set of illumination example images (of other faces captured under different illumination) and a facial expression motion sequence, we aim to generate novel expression sequences of the input face under the lighting conditions in the illumination example images. The input image and illumination example images can be taken under arbitrary unknown lighting. In this paper, we propose two methods in which a 3D spherical harmonic morphable model (SHBMM) can generate images under new lighting conditions with remarkable quality even if only one single image under unknown lighting is available, not only for static poses but for dynamic sequences as well where the face is undergoing subtle high-detail motion.

Publications

A Hierarchical Framework For High Resolution Facial Expression Tracking

[publications]

We present a novel hierarchical framework for high resolution, nonrigid facial expression tracking. The high quality dense point clouds of facial geometry moving at video speeds are acquired using a phase-shifting based structured light ranging technique. To use such data for temporal study of the subtle dynamics in expressions and for face recognition, an efficient nonrigid facial tracking algorithm is needed to establish intra-frame correspondences. In this paper, we propose such an algorithmic framework that uses a multi-resolution 3D deformable face model, and a hierarchical tracking scheme. This framework can not only track global facial motion that is caused by muscle action, but fit to subtler expression details that are generated by highly local skin deformations. Tracking of global deformations is performed efficiently on the coarse level of our face model with one thousand nodes, to recover the changes in a few intuitive parameters that control the motion of several deformable regions. In order to capture the complementary highly local deformations, we use a variational algorithm for non-rigid shape registration based on the integration of an implicit shape representation and the Free Form Deformations (FFD). Due to the strong implicit and explicit smoothness constraints imposed by the algorithm, the resulting registration/deformation field is smooth, continuous and gives dense one-to-one intra-frame correspondences. User-input sparse facial feature correspondences can also be incorporated as hard constraints in the optimization process, in order to guarantee high accuracy of the established correspondences. Extensive tracking experiments using the dynamic facial scan of five different subjects demonstrate the accuracy and efficiency of our proposed framework.

Publications