How do I create an automated lip-sync?

Automating a lip-sync in Harmony can help to save a lot of time when animating. While the automation isn’t always perfect, it sets a strong baseline that you can easily edit. 

This article covers the process of setting up an automated lip-sync in Harmony.

Step 1: Preparing the artwork

Before automating the lip-sync, the artwork for the mouth must be prepared as follows:

• All mouth artwork should exist on a single, separate layer. Make sure to name the layer something that you will easily recognise.
• Each mouth drawing should correlate with one of the mouths in the Toon Boom mouth chart. Renaming each of the drawings to match will make the mapping of the lip-sync easier. You can do this using the Drawing Substitutions view.

• If you have mouths with multiple emotions and views, use naming conventions to differentiate between them. For example, a happy mouth in the 34 front view could be named ‘34F_HA_C’.

Step 2: Import and detect the audio

Before you can map the lip-sync, you have to import and detect the audio so that Harmony can analyze the sounds used.

1. In the top menu, select File > Import > Sound… and choose your sound file.
2. The audio will be imported as a new layer in the Timeline view.

3. To detect the audio, do one of the following:

   • In the Timeline view, right click anywhere on the audio layer and choose Lip-Sync > Auto Lip-Sync Detection.

     

    

   • In the Xsheet view, right click on the audio column and select Lip-Sync > Auto Lip-Sync Detection.

4. The audio will be detected.

Step 3: Map the lip-sync

After detecting the sound, you can now tell Harmony what mouth shapes to assign to which sound.

1. To open the Lip-Sync Mapping window, do one of the following:

   • In the Timeline view, right click anywhere on the audio layer and choose Lip-Sync > Map Lip-Sync...

     

      

   • In the Xsheet view, right click on the audio column and select Lip-Sync > Map Lip-Sync...

     

2. The Lip-Sync Mapping window opens:

3. In the Audio field, you will see the name of the sound layer that you launched the window from. If this is not the sound layer you want to map, simply close this window and relaunch it from the correct audio layer.
4. In the Destination Layer field, use the dropdown to select the layer that you want to apply the mapping to. It is recommended that you name all layers accordingly before mapping so that you can easily find the one you need.
5. In the Symbols in Layer field, you will only need to make a selection if the drawings you want to use are in a symbol. If not required, leave as is.
6. Under the Mapping section, the left-hand side shows the mouth chart names, while the right-hand side needs to contain the names of your mouth drawings. You will need to designate specific mouth drawings to certain mouth shapes in the mouth chart. Insert the name of each drawing that should correspond with the mouth in the chart. If you named your drawings to match the chart beforehand, this step should be fairly simple.
7. Click OK once done.
8. Playback the animation to view the lip-sync.

Step 4: Manually adjust the lip-sync

Depending on the result of the automated lip-sync, you may want to adjust a few areas by manually switching drawings:

1. Activate the Scrubbing button in the Playback toolbar to hear individual sounds as you scrub.
2. Find the mouth layer and identify areas in the lip-sync that need attention.
3. On the mouth layer, manually switch the mouth drawings by selecting the drawing frame you want to change and doing one of the following:

   • In the Drawing Substitutions view, find the mouth you want to switch to and select it.

     

   • In the Library view, use the Thumbnail slider in the top left to scroll through the available drawings.

     

   • In the Timeline view, use the square brackets on the keyboard ( [ and ] ) to switch to the next or previous drawing.

Q&A 

Question: 

Why didn’t my lip-sync mapping work?

Answer: 

Check the following:

• That the audio layer was detected.
• That all required fields in the Lip-Sync Mapping window are filled correctly. Make sure you are mapping the right audio layer, have chosen the correct destination layer, and that the mapping fields have the names of the actual drawings correlating with the chart names. If your mouth drawings are named ‘1’, ‘2’, ‘3’, etc. then those are the names that must be input.
• That you are not playing the scene in the Render view mode. 

Question: 

My lip-sync worked, but why doesn’t it look right?

Answer: 

1. The lip-sync result can be affected by the following characteristics in audio:
   • Low quality sound files can ‘smudge’ the audio, making it difficult for Harmony to detect specific sounds.
   • Very fast dialogue can result in ‘flapping’ mouth shapes.
   • Slurred speech can make it difficult for Harmony to detect specific sounds.
   • Although languages other than English can have a good result, languages with very unique phonemes may result in Harmony trying to match to the closest guess. In this case it’s best to manually match those mouth shapes.
2. Make sure that the names used in the lip-sync mapping were correct and spelled exactly the same as the drawing’s name.

Question: 

Why can’t I hear the sound when I play back my scene?

Answers: 

1. Make sure the Sound button is activated in the Playback toolbar.
2. Check your speakers, headphones, and volume.