Motion Capture in Face Robot
The human face has a finite number of useful landmarks to capture for animation. The areas between these landmarks are only reacting to the movement of these points, albeit in a very complex manner. Face Robot is particularly useful for motion capture animation because it requires only a small number of facial capture markers (~32) to achieve high quality results.
When you’re doing the motion capture session, you simply place markers on the critical areas of the face. Then in Face Robot, you tune the soft tissue to accurately move the face in response to the captured motion. The results are more subtle movements and believable animation, all facets which you can control after the capture session has ended.
For more information in general about getting motion capture set up, see Motion Capture.
For information about working with motion capture in Face Robot, see these sections:
- Preparing the Mocap Data for Retargeting (C3D Files)
Motion Capture Overview
This is a basic overview of the motion capture workflow with Face Robot.
1. Place the mocap markers on the face of the human model.
Face Robot is driven by 48 controls called the Animation Control Set (including the "lookat" control and the fine controls around the eyes). A subset of these controls, called the mocap control set (32 control points), can be driven directly by motion capture data, so the motion capture marker placement is related closely to the positions of these controls. In addition, three stablization motion capture markers must be used to define the global motion of the head (the head control).
If you place the markers on the model’s face as shown in this image, the mocap data will map nicely to the animation control set in Face Robot. Make sure to name the markers with an appropriate naming convention that maps to Face Robot (see Animation Control Set).
2. Record the desired facial mocap expressions.
Make sure to take the important poses: base, key, extreme expressions, range of movement, standard expressions, lip sync “visimes�
The expressions you need to capture obviously depend on the performance that is required for the project. However, experience has shown that at least the following captures should be made to facilitate the tuning of the head in Face Robot:
- Base or zero pose: There should be a capture of a facial expression that corresponds to the rest pose 3D scan. The actor’s face should be in a pose with eyes open, mouth closed, and expressionless. This simplifies the retargeting in the same way a T-pose is used for a default body pose in character animation.
- Range of Motion: The range of motion should exercise the widest possible range of facial deformations, from smiles and screams to frowns and sneers.
- Key Poses: In particular for 3D scanned heads, it is also useful to capture key poses because they make it easier to align the motion capture data with the facial model. This can be done either as part of the range of motion or in separate takes.
See Facial Scanning and Reference Photography for examples of what kind of poses to take.
3. Clean up and filter the data, saving it as C3D data. This format is a standard optical marker-based file format used predominantly for facial capture.
4. Organize the mocap data into folders for each capture session.
5. In Face Robot, prepare the mocap data from each mocap session for the face in Face Robot. This involves loading the zero (base) pose C3D file into Face Robot and creating name and face map files. See Preparing the Mocap Data for Retargeting for how to do this. A video clip is included to help you get going.
Once you’ve done this, the mocap data is ready to be used at any time and on any character in Face Robot.
6. Apply the mocap data to the animation controls on the face in Face Robot, including static poses (see Applying the Retargeted Mocap Data). A video clip is included to help you get going.
7. Retarget the mocap data to the face in Face Robot by adjusting and calibrating it. You can also set offsets on the animation controls on top of the mocap data (see Adjusting the Retargeted Mocap Data). A video clip is included to help you get going.
8. Bake the resulting animation into plotted fcurves (see Plotting the Mocap Data).
Tips for Doing Mocap with Face Robot
- The placement of the stabilization markers is important. The nose bridge marker must be placed such that its movement relative to facial expressions like "scrunching" the nose is minimal. Likewise, the left and right forehead markers must be placed such that their movement relative to eye brow motions are minimal.
- One model, one set of markers: Save all C3D files in one session with the same model and markers. Then when you’re retargetting the data, you only need to later create one name map and one face map to apply to all C3D files taken in that session.
- Facial scans with mocap markers: To help you figure out which marker is which when you load the C3D file into Face Robot to create a name map, you should take a scan of the model with the markers to act as a reference.
- To speed up interaction, press F2 and select Toggle Face. You can quickly scrub to the frame of the reference video that you want to work on and the click Toggle Face again.
- To make a preview, click on the camera icon at the top of the viewport and choose Start Capture. This creates a flipbook using whatever view mode is shown in that viewport. See Previewing Animation in a Flipbook for information.
Adjusting Mocap
- You can control the amount of motion coming from each mocap marker by choosing Adjust from the Settings tab on the Act panel.
- Make sure that the Head Controls > Enable Rotation option is not selected for stabilized motion.
- These marker settings have given the best results so far:
- Global Controls > Ears - 0.3
- Special Controls > LCheekIn - 0.2
- Special Controls > RCheekIn - 0.2
- Markers Controls > NoseTip - 0.5
See Adjusting the Retargeted Mocap Data for more information.
About the Animation Controls
All of the animation controls are animatable, but you generally only need the yellow ones. If you work in Local space (as opposed to Global or View), you will notice that the controls are always at 0,0,0, even when they are moving with the mocap. This is great for getting back to the neutral state once you start keyframing on top of the mocap. You can press Ctrl+Shift+r to return to the neutral pose. It's a good idea to key back to a neutral pose as often as possible when layering in animation. This way, the mocap won't be pulled off-model by mistake.
Head Not Moving?
If the head isn't moving and it should be, it's probably because the head translation and rotation are disabled in the Adjust property page by default. Simply activate them (see Adjusting the Retargeted Mocap Data) and set the translation scale to 1 to get the full head translation.
If the head still doesn't move, it could be that the mocap data you are using is stabilized, meaning that the head translation and rotation have been filtered out of the mocap clip. Use the unstabilized version if you want head motion.
The last possibility is the mocap take has very little head motion: check your original mocap clip to confirm this.
Making Curves for the Mocap Data
You will notice that there are no visible keys or fcurves for the mocap. This is because mocap is in a live retargetting mode until you bake it (plot it) into fcurves. This lets you first make the adjustments to the mocap data.
When you plot the motion capture data, the fcurves often have many keys, usually one per frame. A high-density fcurve is difficult to edit because if you change even keys, you have to adjust many other keys to retain the overall shape of the curve.
See Plotting the Mocap Data and Keyframe Animation for more information.
Check the Jaw
Because there is no good way to capture the jaw motion with the skin sliding around and affecting the markers, you may need to make adjustments to the jaw control in order to compensate and keep the teeth the proper distance apart. You can do this simply by keying the translation of the jaw control. Remember to key back to a neutral pose at some point. Use a video reference and check the extreme poses and anywhere that the teeth are supposed to be touching.
Fine-tune the Mouth Shapes
The most important parts of lip sync can't really be captured, so most of your time will be spent addressing this. The interior shape of the lips, the way the fat bunches up in the corners, the way the skin sticks together ... this stuff makes or breaks realistic lip sync. The best way to get it right is to model the fixes with localized shape animation.
shape animation under construction - use the shape manager
For more information, see Overview of the Shape Manager (http://softimage.wiki.avid.com/xsidocs/shaped2.htm).
Preparing the Mocap Data for Retargeting
Retargeting allows you to use motion capture data on any size or proportion of face. This is especially useful when you want to animate different characters with the same movements and expressions that you've saved in mocap data files.
Face Robot uses C3D data files for motion capture animation, which is a standard optical marker-based file format used predominantly for facial capture. The C3D format is a public domain, binary file format that is used in Biomechanics, Animation, and Gait Analysis laboratories to record synchronized 3D and analog data. It is supported by almost all 3D Motion Capture System manufacturers, as well as other companies in the Biomechanics, Motion Capture, and Animation Industries. For more information, see: c3d.org (http://www.c3d.org/c3d_introduction.html)
Before you can use the mocap data from these C3D files on a solved head in Face Robot, you must prepare the data from each mocap session. This involves loading the zero (base) pose C3D file into Face Robot and creating name and face map files. These files map the mocap data to the face in Face Robot.
You must prepare the mocap data in Stage 1: Assemble when your scene is empty. Once you’ve prepared the mocap data, it’s ready to be used at any time and on any character in Face Robot.
To get you started, you can download this zip file to watch a video about how to prepare motion capture data for Face Robot: Preparing Mocap Data
Creating Name Maps and Face Maps
- A name map (nmap) creates links between the marker names in the C3D mocap data file and the corresponding control names on the solved head in Face Robot, like a name-mapping template.
- A face map (fmap) provides a reference of the proportion of the face of the human model from whom you captured the mocap data. The face map takes a snapshot of the position of the mocap markers on the human model in a neutral pose (expressionless) which allows Face Robot to map that data onto a solved head. You cannot retarget mocap data without a face map.
When you do a mocap session with the same human model and markers, the marker names will be the same for all C3D files captured in that session and the face proportions will all be the same. This way, you need to create only one name map and one face map that will apply to all the C3D files from that session set. The only requirement is that all the motion capture files exist in the same folder as the generated face map. If so, Face Robot then uses a single fmap to calibrate all the mocap data in the folder.
If you are using the same actor but across different mocap sessions (so the markers are removed and reapplied in between), you will need to generate a seperate fmap for each session. This is to account for any slight position shifts of the markers between sessions. As well, if you are using different actors, each one would require their own face map for calibration.
It is probably best if the person who placed the mocap markers on the model or supervised the mocap marker placement does the name mapping in Face Robot. This is because they will know the correlation between the mocap markers and the animation controls in Face Robot. This person usually creates a cheat sheet of the marker placement on the model to help them (or somebody else) figure out which marker is which when the mocap data is loaded into Face Robot.
To Create a Name Map
1. Start with an empty scene in Face Robot.
On the Stage 1: Assemble panel, click the Tools > Face tab and then click the Mocap button.
2. In the Facial Mocap Tools property panel that appears, click the Load C3D button to load a mocap file. This should be the zero or base pose that has no particular expression, which makes retargeting the mocap data easier to do.
A set of blue ball-shaped markers in the shape of a head is loaded into the viewport. Each of these markers represents a mocap marker that was on the model’s head during the capture session.
3. Click the Create Name Map button in the Facial Mocap Tools property panel.
This brings up a Name Map Wizard property panel with a list of all the Face Robot animation controls (buttons) to which you must pick the corresponding marker in the mocap data. These correspond to the naming convention that you used when you created the mocap files: you should have corresponding markers for each animation control.
If you have an existing name map file you want to use, click the Load button in this property panel to open the browser and load the file.
| If you use the same naming conventions for the mocap markers as described in Motion Capture Overview, then you can use the following name map file to get a quick start. Download and unzip this name map file, then load the embedded name map file into the Name Map wizard.
|
4. Select a blue ball marker on the C3D data in the viewport, then click the corresponding animation control button on the page.
The picking session uses symmetry so that you only have to select the marker on one side of the face for markers that have a left and right equivalent.
| The Global Motion controls map to the nose bridge, and left and right forehead mocap markers. These markers represent the head’s general (global) rotation and translation movements.
If you’re using stabilized (extracted) mocap data, use the buttons on the Extracted Motion tab. See Using Extracted Motion Data (Stabilized Mocap) below for information.
|
5. When you’re done, click the Save button in the property page to save the name map (.nmap) file in the same folder as you have the C3D files.
Now you need to create a face map, which takes a snapshot of the proportion of the human model’s face in a neutral pose (expressionless) so that it can be mapped onto a solved head in Face Robot.
To Create a Face Map
6. Make sure that the C3D file you’re using is the base pose.
7. Move the timeline cursor to the frame where the markers are closest to the neutral pose (expressionless).
The neutral pose need not be at the start of the animation sequence: simply find any frame in which there is a good expressionless pose.
8. In the Facial Mocap Tools property panel, click the Create Face Map button to generate a face map using this pose.
9. Enter a name for the face map file (.fmap) and save it in the same folder as the name map and C3D files.
Using Extracted Motion Data (Stabilized Mocap)
The mocap tools in Face Robot handle the stabilization of mocap data (extract the head motion from the facial marker motion), but you may choose to use other stabilization techniques (software that cleans up mocap data typically offer this option). In this case, the C3D file that is brought into Face Robot will have the facial markers showing just the facial expressions (the head motion will have been extracted), while the actual head rotation and translation are represented by some additional markers in the file offset from the facial markers.
If you are working with this type of mocap data, you can create a name map as described previously, except that you click the Extracted Motion tab in the Name Map Wizard property page and map only the Global Motion nose bridge, left forehead, and right forehead controls on this tab to the corresponding mocap markers. You can then create a face map in a base pose as usual.
Applying the Retargeted Mocap Data
You can load and apply mocap data to the animation controls on a solved head in Face Robot. You may need to calibrate the data from the motion file to have it properly fit the proportions of the head to which you’re applying it — see Calibrating the Mocap Data for more information.
To apply a motion file to a solved head
1. Click the Library > Motion tab on the Act panel and select the appropriate motion file from the list.
If you need to, click the browse (...) button to go to a folder where your motion files are located.
2. Play back the mocap data to see how the animation fits the face.
You will probably need to make some adjustments, as well as calibrate the mocap data. See Adjusting the Retargeted Mocap Data for more information.
Removing the Retargeting Data
You can remove the retargeting operator and associated data (name map and face map files) from a face’s controls. This puts the face back into its neutral pose as it was before you applied the mocap data.
To remove the retargeting data
- On the Act panel, choose Library > Settings > Clear Retarget.
Applying and Creating Poses
In addition to motion files, you can also load a static pose for the head. Static poses are useful for testing out expressions on the face, such as extreme poses, as well as for easily returning to a base pose.
A pose is a static source that contains no animation and is saved in a normalized format so that it can be applied to any solved head. As a result, you may need to calibrate the data from the pose to have it properly fit the proportions of the head to which you’re applying it — see Calibrating the Mocap Data for more information.
You can save poses as pose presets that you can apply, or save them in static action sources that you can load in the animation mixer.
To get you started, you can download this zip file to watch a video about how to apply poses and create your own pose presets: Applying and Creating Poses. This also includes information on calibration.
To load a pose
1. Click the Library > Pose tab on the Act panel.
2. Select a pose from the list and click Apply.
If you need to, click the browse (...) button to go to a folder where your pose files exist.
Creating Pose Presets
After you’ve moved some controls on the face into specific expressions, you can save those expressions in a pose preset. These files will then appear on the Library > Pose tab. This way, you can create an entire library of pose presets for a face.
To save a pose preset
1. Move the animation controls on the face to create a specific expression.
2. On the Face > Select tab on the Act panel, click the Pose All button to save a pose using all controllers on the face.
You can also select certain controllers and click the Pose Sel button to save only those controllers as part of the preset. For example, select and position only the controls around the mouth, then click Pose Sel to save a pose that affects only the mouth.
3. In the browser that appears, select a folder and give a name to the pose preset.
The pose preset appears on the Library > Pose tab (you may need to refresh the tab to see it).
Creating Action Source Poses
In addition to pose presets, you can save the poses in an action source, which allows you to use the animation mixer. A pose is static action source, which is a “package� of animation. You usually create action sources to contain animation, but you can also use them to store a pose, which contains no animation.
Once you have created an action source, you can apply it on the same model or share it with a different model. You can create an entire library of actions and share them among any number of models. You can also load it into the animation mixer as an action clip where you can manipulate it in a number of ways.
For information on using actions in the animation mixer, see About Action Sources and Clips (http://softimage.wiki.avid.com/xsidocs/actions_AboutActionSourcesandClips.htm). You can also open the animation mixer (press Alt+zero) and click the help (?) icon in the animation mixer toolbar to get information on using the mixer.
To save an action source pose
1. Choose File > Preferences from the main menu bar and click the Custom > Face Robot icon in the Preferences property editor. Select the Use Mixer Poses option.
2. Click the Pose All button on the Face > Select tab to save a pose using all controllers on the face.
You can also select certain controls and then click the Pose Sel button to save only those controls as part of the preset. For example, select and position only the controls around the mouth, then click Pose Sel to save a pose that affects only the mouth.
3. In the Store Action dialog box that appears, give a name to the pose and set the number frames over which you want the pose to last (the Default In and Out values).
If you want to add the pose directly to the animation mixer, select the Add Source as a Clip in the Mixer option.
You can also set the Clip Offset, which is the frame at which the clip starts in the mixer.
The poses are stored as action sources that you can find in the Face model’s Mixer > Sources > Animation folder (open an explorer to find them).
Calibrating the Mocap Data
After you have applied the mocap data for the face in Face Robot, you’re ready to calibrate the mocap data. Calibrating means adjusting the mocap data that was created for a human model to fit the proportions of the face you have loaded in Face Robot. This way, the facial movements captured in the mocap data will map properly to create the correct expressions on the face in Face Robot.
The calibration data is saved for the head, so you need to calibrate the data only once and then it is calculated when you map other mocap data files to this head.
To calibrate the mocap data
1. Apply the mocap data and run it through to see where there are problems with the proportions of the face.
You will probably want to go to frames where the face is in an extreme pose (such as a big scream or a yell) to find problems.
2. On the Act panel, choose Library > Settings > Calibrate to open the Calibrate property page.
3. On this page, set the Scale and Pitch for each of the areas on the Up and Down Direction tabs: Upper lip, Lower lip, Jaw, Nose, and Eyes.
- Pitch is set in degrees.
- Scale is the scale factor (1 is 100%) of how the normalized data is applied to the face.
|
| In the case of a yell or scream pose, you would probably need to adjust the Scale of the Jaw and Lower Lip settings on the Down Direction tab.
|
Adjusting the Retargeted Mocap Data
After you have applied the mocap data for the face in Face Robot, you’re ready to adjust it to fit the proportions of the face. The data from the mocap files drives the animation of the controls on the face in Face Robot.
To get you started, you can download this zip file to watch a video about how to retarget the motion capture data for Face Robot: Retargeting the Mocap Data. This includes offsetting, adjusting, and plotting the data.
To adjust the mocap data
1. On the Act panel, choose Library > Settings > Adjust to open the Adjust property page.
2. On this page, you can:
- Mute the retargeting mocap data: select the Global Controls > Mute Retargeting option.
- Filter high-frequency noise from the mocap: select the Global Controls > Filter Noise option.
- Scale the retargeting mocap data: use the control sliders in the Global Control and Special Controls areas. The scaling of the controls is modified with respect to the neutral pose as defined in the face map (.fmap) file.
- Activate the global rotation on the head: select the Head Controls > Enable Rotation option. You will need to select this option if you’re tuning the neck weighting. Most of the time, however, you won’t need to have the head rotation active. If you’re working with stabilized motion, make sure that this option is not selected.
- Activate the global translation on the head: select the Head Controls > Enable Translation option. This lets you key the head control so that a head attached to a moving body will stay with the body!
- If you select Enable Translation, you can scale the translation of the global motion of the head by setting the Head Controls > Trans Scale slider.
- Adjust the gain for each mocap marker: set the sliders in the Markers Controls area. These sliders determine how much of the mocap retargeting data is applied to the animation control. A value of 0 means that no mocap is applied to that control, while a value of 1 means 100% of the mocap data is applied to the control.
|
| You may notice that the Adjust panel gets cleared and rebuilt every time you apply mocap, or when you clear the mocap. This usually makes sense because the adjustment keys are specific to a particular mocap sequence. However, there are times when you may have made some tweaks to a C3D file and want to refresh the mocap with a newer version of the same mocap. In this case, you want to preserve any keys you have made on the Adjust panel. To do this, there are two simple commands you can use, as described in Troubleshooting.
|
Offsetting the Mocap Data
Once you have applied mocap data to the face, you can key offsets on top of this data to adjust it. The offset is carried through to the mocap data.
To add offsets to the mocap data
1. Select some animation controls and move them to where you want.
2. Set keys for these controls by pressing the K key or by using the buttons on the Stage 5: Act panel. See Keyframing for more information.
3. Plot the animation to bake the mocap data and the offset animation into fcurves (see Plotting the Mocap Data).
Plotting the Mocap Data
You will notice that there are no visible keys or fcurves for the mocap. This is because mocap is in a live retargetting mode until you bake it (plot it) into fcurves. This lets you first make the adjustments to the mocap data before committing it to fcurves.
When you are at a point where you no longer need the retargetting tools, or you just need fcurves to remove pops or make other adjustments, you can plot/bake the mocap into keyframes (fcurves) which you can then edit. Plotting create composite fcurves that contain the mocap and any animation (offsets) that you keyed into the animation controls. These fcurves are applied to the face’s animation controls.
When you plot the motion capture data, the resulting fcurves often have many keys, usually one per frame. A high-density fcurve is difficult to edit because if you change even a few keys, you have to adjust many other keys to retain the overall shape of the curve. Because editing these fcurves is not always easy, there are tools in the animation editor (like the graph editor in Maya) that can help you work with them: the HLE (high-level editing) tool and curve processing tools (smoothing, resampling, and fitting).
If the fcurves are in an action clip in the animation mixer, you can also add a clip effect to the clip to adjust the fcurves without affecting the original animation.
For more information, see Shaping High-Density Fcurves (HLE) (http://softimage.wiki.avid.com/xsidocs/fcurves_ShapingHighDensityFcurves.htm) and Modifying Action Clips with Clip Effects (http://softimage.wiki.avid.com/xsidocs/actions_ModifyingActionClipswithClipEffects.htm).
To plot the mocap data and keyframe animation
1. On the Act panel, choose Library > Settings > Plot to open the Plot property page.
2. On this page, set the Interval Size, which is the interval of keyframes set on the fcurves. For example, if you specify 1 as the frame step, a keyframe is created on every frame on the fcurve for each animation control.
3. Set the Start and End Frames to specify the frame range that you want to plot. By default, the timeline range is used, so the whole animation will be plotted if you don’t change these frame values.
4. Click OK to start plotting.
The plotting process removes the mocap data and retargeting data from the animation controls, then bakes that into fcurves along with any keys that are on the controls. The fcurves of the plotted animation are then applied to the animation controls after the fcurves are fitted.
5. If you want to edit these fcurves, select the controls you want and press the 0 (zero) key to open the animation editor.
For information on editing fcurves in general, click the help (?) icon in the animation editor, or see Overview of the Fcurve Editor (http://softimage.wiki.avid.com/xsidocs/animeditor13.htm).
Using the fr_PlotRetargetedFace Command Line Argument
Instead of using the Library > Settings > Plot command on the Act panel, you can plot using this command line argument to bypass the Plot property page:
fr_PlotRetargetedFace([start frame], [end frame], [interval size]);
The following example plots between frame 100 and frame 200, setting one key every 10 frames.
fr_PlotRetargetedFace(100, 200, 10);
If you specify a null for any of the arguments, an interval of 0, or a start frame that’s less than or equal to the end frame, the Plot property page appears.
