Overview

FACE (Facial Animation Communication Engine) is a real-time facial animation tracking system, which is extracting a set of facial animation control parameters from video input of a human face, and enhancing facial expressions of 3D virtual human models in real-time.

This facial animation tracking system can be integrated into a game engine to link the video motion tracking data to real-time facial animations of a virtual 3D facial model.

Implementation

Figure 1: FACE System Overview

In acquisition stage, the image sequences of a human face are captured by a video camera in 30 fps. FACE requires both grayscale and RGB colour images for detection and tracking the face and its features, therefore, the acquisition module of the FACE system provides both colour and grayscale images to the analysis module. DirectShow API is used to capture video frames in this stage.

In the Analysis stage, a human face is detected and the facial features, such as eyes, eyebrows, nose and lips are also detected from video frames. The detected face and its features are tracked by this analysis module in order to analyze FAPs (Facial Action Parameters). We use a hybrid method, a combination of appearance based face detection, skin colour distribution and edge detection method. The face is initially detected by using Haar-like classifier object detection method, and then the flesh colour of the face skin is sampled and the skin colour distribution is tracked by Continuous Adaptive Mean Shift (camshift).

Once the face is detected, localization of facial features is performed, especially in 2 major subregions; mouth and eye region in order to increase performance of tracking. The facial features, such as eye and lips are localized by using Principal Components Analysis (PCA). Other facial features are extracted by edge detection methods, such as Hough Transform and Canny Edge detection algorithms. And, Kalman filter is used to stabilize the tracked data.

The translation and rotation information of the face and its features are finally analyzed in this stage in order to construct the facial animation control parameters, utilized by existing standards, FAPs.

In the Synthesis stage, the FAPs are applied onto a skeletal-based animated virtual facial model in order to pilot facial animations and to enhance facial expressions. The synthesized animations and expressions are visualized in an OpenGL based Windows application.