Lipsync gets off sync and stops before the end of the audio

What do you need help with?

Question

Installation method

UPM

Issue area

Lip Sync / Facial Animation

Impact

Blocker - I cannot continue

Unity version

6000.3.19f1

Convai Unity SDK version

4.5.0

Character ID

c15f9644-9f35-11f1-80c8-42010a7be02f

Target platform

Meta Quest

Where does it happen?

Both sample and custom scene

Short summary

Lipsync gets off sync and stops before the end of the audio

What happened?

Hello,

I’ve been having an issue with the lipsync on my convai character which is using a GCP german voice.
I am using unity version 6000.3.19f1, i did try to change the language just to see what happends but the problem is still the same, althought the dialogues are in german, but i wanted to test if using only english will solve the issue but it didnt.

The issue is that sometimes the lipsync seems to get offset from the audio, it seems to run untill half and then at the end of the message the lips stop moving before the audio ends.

This is really a blocker for me, where I want the avatar to speak german with complete normal lipsync otherwise character is of no use, if the speech is still going but lips has stopped moving.

What did you expect?

That it will work perfectly with German language

I checked the relevant documentation, resources, and similar forum posts before posting.

on

I confirm that I did not include API keys, tokens, passwords, secrets, private character data, private conversation logs, or customer data.

on

Could you please share a video recording and the relevant logs so we can review the issue properly?

what logs? where i can see them relevent, and more onto the problem i am using tags in the Narrative Design would that be a problem for TTS pipline for lipsync?

Please go to Project Settings → Convai SDK and set Global Log Level to Debug.

Test, then copy all the logs, paste them into a .txt file, and share the file with us.

Same issue here on SDK 4.5.0 (Unity 6000.3.17f1, CC4 rig, `cc4_extended`, native LiveKit transport). We reproduce it in the macOS editor as well as on Quest, so it is not platform-specific. Posting our findings here since this thread is open and asks for logs; the older thread (#6337, “Bug lipsync stops prematurely”) was marked solved after 4.4.0, but 4.4.0 replaced that mechanism with a new one that fails the same way for the user.

Symptom

On long turns the mouth freezes on one pose for a few seconds and then fades to neutral while the audio keeps playing to the end. Short turns (~10 s) never show it. Reproduced 2 out of 4 runs with a fixed 73-second line sent through `ConvaiCharacter.SendNarrativeSpeech`, mic muted. Default LipSync component settings (Latency Mode default, Max Buffered 3 s, Fade Out 0.2 s).

Log of a failing run (LipSync category on Debug)

```
[13:24:19.689][Info][LipSync] TurnStats - Server: 4396 frames | Received: 4396 frames | Match: YES | Turn Duration: 49.14s | Audio Duration: 73.27s | FPS: 60.00
[13:24:20.444][Debug][LipSync] TurnStats closed input before animation boundary: armed sample-progress watchdog.
[13:24:37.024][Debug][LipSync] AudioPlayback: isPlaying=False ... state='Playing' ... sync=audioActive:False,gateOpen:True,clock:SampleLocked
[13:24:37.024][Debug][LipSync] Gate resume: state='FadingOut', indexed=True, audio=True, speaking=True ... afterAudioStartMs:1.3,afterAudioStopMs:6.6
[13:24:41.523][Warning][LipSync] Completion recovery: sample clock stopped progressing; faded visual stream at audio position.
sync=audioActive:True,gateOpen:True,clock:SampleLocked,afterAudioStartMs:4500.6,afterAudioStopMs:4506.0,sample=unavailable
[13:24:43.778][Info][LipSync] Response complete: ... frames=0, recoveredGaps=0, clock=SampleLocked, reason=SampleClockStall

`Character started speaking` was at 13:23:30, so the visual stream was killed ~37 s into a 73 s turn. The audio played on to the end. At the moment of the freeze the diagnostics still showed `buffered=2.98 s` and `stream=73.3 s`, i.e. every frame was there. The LipSync Drift Monitor shows an `audio stop (frozen @0.00s)` / `audio start` / `gate resume` triple every 2–5 s throughout the turn; those line up with the natural pauses between sentences, not with network trouble (`underruns=1`, `overflows=0` on the LiveKit `AudioStream` snapshot for the whole turn).

What we think happens (from the 4.5.0 sources)

1. `AudioStream.UpdatePlaybackState` raises `PlaybackStopped` when the rendered peak stays under `SignalStopThreshold` for `StopTimeoutSeconds = 0.5`, and `PlaybackStarted` when signal returns. Every pause between sentences toggles `CharacterAudioPlaybackStateChanged`.
2. Once `TurnStats` has arrived, `ConvaiLipSyncBridge` has the completion watchdog armed. `TickCompletionWatchdog` requires the audio timeline target to advance; if it does not for `CompletionStallTimeoutSeconds = 0.5f` it calls `ExpireCompletionWatchdog(SampleClockStall)` → `BeginClosing` + `StopSmooth()`. That is terminal: the response is closed and a later clock recovery cannot reopen it.
3. After a signal-level stop/start flip the timeline snapshot can be `sample=unavailable` for several seconds (4.5 s in the run above) even though audio is playing (`audioActive:True`). 0.5 s is not enough budget for that, so the watchdog fires and the mouth dies while the audio continues.
4. A second, smaller path: after `TurnStats`, every `PlaybackStopped` calls `StopSmooth()` with `FadeOutDuration = 0.2 s`; if the pause outlasts 0.5 + 0.2 s the fade completes, `TickFadeOut` does `FullReset()` → `Idle`, and `NotifyAudioPlaybackStarted` only recovers from `FadingOut`, not from `Idle`.

Both need `TurnStats` to arrive while audio is still playing. That is only the case for long turns, because generation is faster than real time (49 s of generation for 73 s of audio here). For a 10 s turn `TurnStats` arrives ~2.5 s before the audio ends, so there is almost no window. That is why this looks like a “long answers” bug.

What we tried

Raising Fade Out Duration to 2.0 s did not help; the freeze-then-fade still happened at ~33 s with the `SampleClockStall` reason. No Inspector setting reaches `CompletionStallTimeoutSeconds`, `StopTimeoutSeconds` or `TargetBufferSeconds` (0.2 s), which are constants.

Suggestions

- Don’t treat a signal-level `PlaybackStopped` as a terminal boundary while the owning response still has un-rendered frames; only a disposed/unsubscribed stream, a cancel, or the audio boundary itself should end the visual stream.
- Give the completion watchdog more than 0.5 s, or pause it while `sample=unavailable` and audio is active.
- Let `NotifyAudioPlaybackStarted` reopen a response from `Idle` while it is still the owner and frames are buffered.

Happy to share the drift-monitor CSVs and the per-frame probe data (audio position, buffered, underruns, mouth blendshape weight at 60 Hz) if useful.

Are you still experiencing the issue?