Recording Audio and Lipsync Animation (New for v3.3)
While an iPhone X can transfer facial image data to CrazyTalk Animator wirelessly, with a microphone on your computer, you can synchronously capture facial expressions along with voice recording through the Facial Mocap panel.
Not only the voice of the actor can be recorded, but also the facial expressions with the lipsync (lip shapes) can be automatically captured and generated at the same time.
The benefits are:
- Recording live audio with the facial capture in one session.
- Recorded audio generates lip-sync data on the fly.
- Blend lip-sync and motion capture to create a cohesive performance.
Synchronous Audio Recording during Facial Capture
- Open the Live Face App on your iPhone. For more information about Live Face, please refer to the Workflow for Facial Mocap section.
-
In CrazyTalk Animator, apply a character and make sure it is selected.
- Click the Facial Mocap
button on the Functional Tool Bar to open the Facial Mocap panel (Animation menu >> Facial Mocap).
- Activate the Record Audio for Viseme Track checkbox. Select the audio input installed on your computer from the drop-down list.
- Speak while you are capturing the facial expressions in Record mode.
- Play back the project to observe the character's facial expressions and lipsyncs.
Refinement and Editing
Open the Timeline panel and click the Face button in order to view the recorded data in their corresponding tracks.
You can find the facial image data stored in the Facial Clip track, and the sound waves and lip-sync visemes stored in the Voice Clip track.
Drag the play head to any desired frames and hear the sound waves. If the lip-synching keys are not transferred well, follow the steps below to modify the viseme key:
- Double-click the viseme key in the Lips track.
- In the following popup Lips Editor dialog, replace the lipsync viseme with another one.
- The lip viseme shape on the character will change accordingly.
Original viseme: WOO Modified viseme: Oh
Please refer to the Facial Clips and Keys and Voice Clip sections for more information about the animation for character heads.