Issues with LipSync and Reallusion in Unity

Original Discord Post by ragnarockfb97 | 2024-07-16 11:10:20

Hello everyone. I’m currently having a few problems with some avatars that we imported from Character Creator. Sometimes, the lip sync won’t work at all and some other times, instead of moving the lips, the character moves his eyebrows. I’m also (sometimes) getting an error saying that the indexes for the mesh renderer are out of bounds, although all characters are created more or less in the same way. Does anyone know how I can solve this?

Images:

Reply by ragnarockfb97 | 2024-07-16 11:18:50

Images:

Reply by k3kalinix | 2024-07-16 11:32:13

Hello <@389522858386128918> :wave:t2:

Reply by k3kalinix | 2024-07-16 11:32:19

Welcome to Convai Community!

Reply by k3kalinix | 2024-07-16 11:33:18

Make sure you have assigned the correct object to the Lipsync component.

Reply by k3kalinix | 2024-07-16 11:33:50

Embedded Content:
Creating Lifelike NPCs in Unity with Reallusion CC4 Characters | Co…
By the end of this tutorial, you will have a working @reallusion Character Creator 4 NPC that you can talk to and interact with in a 3D web environment using Unity 3D. We will create a character using Reallusion’s Character Creator 4 and then import that character directly into Unity. Next, we will connect that character to Convai’s AI brain and…
Link: https://www.youtube.com/watch?v=va-qyFa312g

Reply by ragnarockfb97 | 2024-07-16 12:51:11

I’m having the same issue, with the added problem that the jaw now looks like this:.

Images:

Reply by k3kalinix | 2024-07-16 13:08:02

Did you edit the Avatar file?

Reply by ragnarockfb97 | 2024-07-16 13:16:37

I followed the instructions in the video. I imported the character, removed the jawroot from the avatar file, added the jawroot and tongue1 to the respective fields in the lipsync script (along with the cc_base_body, teeth and tongue) and now it’s doing this

Reply by ragnarockfb97 | 2024-07-16 13:19:13

It is moving its lips, but I don’t know why it’s doing that

Reply by k3kalinix | 2024-07-16 13:31:28

I see, let me tag my teammate.
cc: <@415207243487772673>

Reply by ragnarockfb97 | 2024-07-16 14:39:45

You can see for yourself by downloading the character from here https://drive.google.com/file/d/1UPggUny-QYKsxd71LaYZxNnp1Sz2in_3/view?usp=drive_link

Embedded Content:
Socrate.zip
Link: https://drive.google.com/file/d/1UPggUny-QYKsxd71LaYZxNnp1Sz2in_3/view?usp=drive_link

Reply by raymondedmund | 2024-07-16 17:55:14

<@433610960725344267> This seems like a weight issue in the Lipsync. Can you please advise <@389522858386128918> on how to update these values in script?

Reply by ragnarockfb97 | 2024-07-17 13:19:14

Thank you for your help, I’ll patiently wait for your response

Reply by ragnarockfb97 | 2024-07-17 15:26:58

I’ve noticed that if I don’t remove the JawRoot bone from the avatar rig, the mouth stays in place, although it obviously doesn’t move the jaw when it’s talking (although it does move the lips). There is also the issue that it seems that if the model has a moustache, the upper lip doesn’t seem to move

Reply by ragnarockfb97 | 2024-07-17 15:27:49

Attachments:

Reply by ragnarockfb97 | 2024-07-17 15:31:41

Attachments:

Reply by raymondedmund | 2024-07-17 17:40:20

<@389522858386128918> Can you schedule a call with me on my calendar at https://calendar.app.google/FTm4JbSxu6giKJH5A

Thanks a lot

Embedded Content:
Appointments
Link: https://calendar.app.google/FTm4JbSxu6giKJH5A

Replying to ragnarockfb97’s Message

Reply by ragnarockfb97 | 2024-07-17 15:26:58
I’ve noticed that if I don’t remove the JawRoot bone from the avatar rig, the mouth stays in place, although it obviously doesn’t move the jaw when it’s talking (although it does move the lips). There is also the issue that it seems that if the model has a moustache, the upper lip doesn’t seem to move

Reply by raymondedmund | 2024-07-17 17:51:21

Because of Reallusion’s bone structure organisation, we use the lower jaw bone, otherwise the character talks and “lip syncs” without actually opening the mouth

I am not exactly sure why the mustache issue is happening. My best guess is that the mustache is a separate entity not attached to the upper lip blendshapes. We can check by seeing if the Hierarchy of the model in Unity has the mouth and mustache separately. If that is indeed the case, we can check if mustache has the necessary blendshapes for appropriate lip sync, and then modify the script accordingly

Replying to ragnarockfb97’s Message

Reply by ragnarockfb97 | 2024-07-17 15:31:41

Reply by raymondedmund | 2024-07-17 17:55:25

This is because the script updates the Jaw every frame, so even though you are updating the jaw directly, the lip sync script is updating it the next frame and your changes are not showing up