Coding Tutorial
This motion capture tutorial demo is implemented in two independent parts. The first captures and tracks the position of facial markers, and is implemented using MATLAB since it has a powerful image processing toolbox. The second is a simple Processing GUI that draws a 2D face emulating the expressions of the user. The Processing sketch talks to the Matlab app using a TCP socket.
- Use a machine with Matlab and Processing installed (if not already).
- Download the app source code, and unzip its contents to a suitable location.
- In Matlab, Start faceTrack.m
- A sketch will appear that indicates expected marker placement; place the markers as indicated.
- The app is hard-coded to use the color green for markers.
- Start the processing sketch, drawFace.pde when ready. As soon as this is done, Matlab face detection and processing face rigging should start.
- That’s it! Make a few faces to test it
Tutorial
This subsection provides an outline on how to write a simple motion capture and rigging application.
The Processing sketch (a Java-based graphics environment) primarily uses curves and basic 2D shapes to draw the animated face, utilising beginner-friendly functions that are well documented here.
Matlab is a powerful tool when it comes to writing image processing applications. This tutorial makes use of some basic Matlab APIs, and highlights the specific workflow methods we discovered were necessary to obtain a clean stable capture from consumer-grade webcams and non-ideal lighting conditions.
- To detect a set of markers, the original image needs to be filtered to only pick up the pre-defined color of the markers.
- Since an image would be represented in Red, Green and Blue matrices, it is easiest to pick a marker of one of these colors. If this is the case, then it’s a matter of picking up the corresponding matrix. This would provide a grayscale image representing only the chosen color part of the image.
- If another color is to be used, look at rgb2hsv matlab function. HSV is more human method of color representation, and would be easier to pick-up an arbitrary color.
- For marker detection purposes, we need to reduce the grayscale image we have, in which each pixel is represented with 8 bits, to a Boolean one, where a pixel is either black or white. im2bw() Matlab function provides that; a threshold is provided to the function to set the degree of sensitivity.
- The result image may have some noise, which would consist of small green arrays in the picture. Since we do not wish to detect those, a filtering mechanism is needed. bwareaopen() Matlab function will clear out any connected components less than a certain number of pixels in size.
- bwlabel() function can be used to generate an matrix where each connected component is represented with a unique number.
- regionprops() function would use the labeled image to generate a list of connected components, with lots of useful information, such as object centorids and bounding boxes. The required parameters are passed to the function.
- You now have a list of markers. The next step is to determine how a correct set of markers is detected, and then, what face is expressed. This can be done in many ways. The following is a brief description of faceTrack implementation:
- Verification of correct marker detection is done using the number of detected objects, sizes and their relative locations to each other.
- Facial expressions are detected by measuring the distances between markers, and comparing them against the neutral value. The neutral value is stored the first time a face is detected. Also, a normalization is applied to correct against the distance of the face from the camera.