Character sometimes doesn't speak

Core Features Language and Speech
1

We have a very simple implementation of the embedded player that worked well just a few days ago. Today the character has stopped speaking entirely. Sometimes we get lucky with one session after a lot of reloads when it works for a while, but after a while, even that session breaks.

Is anyone else experiencing this issue?

I can see that the messages get received and they also appear in the chat window. Here are the logs:

index.js:2 DEBUG URL
index.js:2 Disconnected
index.js:2 Reconnect logs false true false Disconnected
convai-embed.js:73 isLoading true
index.js:2 White event origin REDACTED
index.js:2 DEBUG URL
index.js:2 Reconnect logs false true false Loading
index.js:2 domain response: Response {type: ‘cors’, url: ‘https://api.convai.com/xp/stream/validateDomain’, redirected: false, status: 200, ok: true, …}
index.js:2 Unexpected content type: text/html; charset=utf-8 but proceeding with JSON parsing
(anonymous) @ index.js:2
(anonymous) @ index.js:2
(anonymous) @ index.js:2
a @ index.js:2
Promise.then
l @ index.js:2
(anonymous) @ index.js:2
Xt @ index.js:2
(anonymous) @ index.js:2
tg @ index.js:2
Ng @ index.js:2
Rg @ index.js:2
(anonymous) @ index.js:2
(anonymous) @ index.js:2
gv @ index.js:2
(anonymous) @ index.js:2
(anonymous) @ index.js:2
x @ index.js:2
N @ index.js:2Understand this warning
index.js:2 Domain validation for session REDACTED: VALID
index.js:2 Transaction ID: REDACTED
index.js:2 DEBUG URL
index.js:2 Whitelisted domain with referrer REDACTED
2index.js:2 DEBUG URL
index.js:2 Original Resolution: 1920x1080
index.js:2 Scaled Resolution: 1920x1080
index.js:2 [Info] - Data channel cannot send yet, skipping sending message: Command - 51,50,0,123,0,34,0,82,0,101,0,115,0,111,0,108,0,117,0,116,0,105,0,111,0,110,0,46,0,87,0,105,0,100,0,116,0,104,0,34,0,58,0,49,0,57,0,50,0,48,0,44,0,34,0,82,0,101,0,115,0,111,0,108,0,117,0,116,0,105,0,111,0,110,0,46,0,72,0,101,0,105,0,103,0,104,0,116,0,34,0,58,0,49,0,48,0,56,0,48,0,125,0
index.js:2 [Violation] Added non-passive event listener to a scroll-blocking ‘wheel’ event. Consider marking event handler as ‘passive’ to make the page more responsive. See
registerHoveringMouseEvents @ index.js:2
registerMouse @ index.js:2
setMouseInputEnabled @ index.js:2
handlePostWebrtcNegotiation @ index.js:2
handleWebRtcOffer @ index.js:2
(anonymous) @ index.js:2
i.emit @ index.js:2
Le.e.onMessage @ index.js:2
handleOnMessage @ index.js:2
webSocket.onmessage @ index.js:2
index.js:2 [Log] - Set video source from video track ontrack.
index.js:2 [Log] - Created new audio element to play separate audio stream.
index.js:2 Connected to Pixel Streaming
index.js:2 Original Resolution: 1920x1080
index.js:2 Scaled Resolution: 1920x1080
index.js:2 [Info] - Data channel cannot send yet, skipping sending message: Command - 51,50,0,123,0,34,0,82,0,101,0,115,0,111,0,108,0,117,0,116,0,105,0,111,0,110,0,46,0,87,0,105,0,100,0,116,0,104,0,34,0,58,0,49,0,57,0,50,0,48,0,44,0,34,0,82,0,101,0,115,0,111,0,108,0,117,0,116,0,105,0,111,0,110,0,46,0,72,0,101,0,105,0,103,0,104,0,116,0,34,0,58,0,49,0,48,0,56,0,48,0,125,0
favicon.ico:1 GET REDACTED 404 (Not Found)Understand this error
index.js:2 [Log] - Received new ToStreamer protocol. Updating existing protocol…
index.js:2 [Log] - Received new FromStreamer protocol. Updating existing protocol…
index.js:2 [Info] - -AllowPixelStreamingCommands=false, sending arbitrary console commands from browser to UE is disabled.
index.js:2 [Info] - using URL parameters true
index.js:2 [Log] - Received input controller message - will your input control the stream: true
index.js:2 [Log] - Received quality controller message, will control quality: true
index.js:2 Received message from UE: RequestDeviceType()
index.js:2 Received message from UE: RequestParams()
index.js:2 DEBUG URL
index.js:2 DEBUG URL
index.js:2 READING: sessionid=REDACTED ?sessionid=REDACTED
index.js:2 Received message from UE: ToggleVisibility({“show”: false})
index.js:2 Unhandled UE command: ToggleVisibility({“show”: false})
index.js:2 Received message from UE: ToggleLoadingScreen({“show”: true})
index.js:2 [Info] - -AllowPixelStreamingCommands=false, sending arbitrary console commands from browser to UE is disabled.
index.js:2 [Info] - using URL parameters true
index.js:2 [Log] - Received quality controller message, will control quality: true
index.js:2 [Info] - -AllowPixelStreamingCommands=false, sending arbitrary console commands from browser to UE is disabled.
index.js:2 [Info] - using URL parameters true
index.js:2 [Log] - Received quality controller message, will control quality: true
index.js:2 Original Resolution: 1920x1080
index.js:2 Scaled Resolution: 1920x1080
index.js:2 Original Resolution: 1920x1080
index.js:2 Scaled Resolution: 1920x1080
index.js:2 Received message from UE: OnChangedTimeOut(0)
index.js:2 Received message from UE: OnConfigChanged()
index.js:2 Received message from UE: OnConnectionStateChanged({“server”: “Disconnected”,“avatar”:“Disconnected”})
index.js:2 Unhandled UE command: OnConnectionStateChanged({“server”: “Disconnected”,“avatar”:“Disconnected”})
index.js:2 Received message from UE: ToggleLoadingScreen({“show”: false})
2index.js:2 DEBUG URL
index.js:2 Reconnect logs false false false Loading
convai-embed.js:73 isLoading false
index.js:2 Original Resolution: 1920x1080
index.js:2 Scaled Resolution: 1920x1080
index.js:2 Received message from UE: OnConfigChanged()
index.js:2 Received message from UE: OnChangedTimeOut(0)
index.js:2 Received message from UE: OnConnectionStateChanged({“server”: “Disconnected”,“avatar”:“Disconnected”})
index.js:2 Unhandled UE command: OnConnectionStateChanged({“server”: “Disconnected”,“avatar”:“Disconnected”})
index.js:2 Received message from UE: OnConfigChanged()
index.js:2 Received message from UE: OnChangedTimeOut(0)
index.js:2 Received message from UE: OnConnectionStateChanged({“server”: “Disconnected”,“avatar”:“Disconnected”})
index.js:2 Unhandled UE command: OnConnectionStateChanged({“server”: “Disconnected”,“avatar”:“Disconnected”})
index.js:2 Received message from UE: ToggleLoadingScreen({“show”: false})
index.js:2 DEBUG URL
index.js:2 Received message from UE: ToggleWebCam({“enable”: false,“port”:“”})
index.js:2 DEBUG URL
index.js:2 Original Resolution: 1920x1080
index.js:2 Scaled Resolution: 1920x1080
index.js:2 Received message from UE: ToggleWebCam({“enable”: false,“port”:“”})
index.js:2 DEBUG URL
index.js:2 Received message from UE: ToggleWebCam({“enable”: false,“port”:“”})
index.js:2 DEBUG URL
index.js:2 Received message from UE: ExpConfig({“device”:“VMob”,“camera”:{“fOV”:38.117530822753906,“panUpDown”:-0.0563201904296875,“panRightLeft”:-4.9518852233886719,“zoom”:-1.334381103515625,“tilting”:-5},“webCam”:{“visionInput”:false,“webcamPosition”:“WebCam UP Left”,“webcamSize”:0.75},“chat”:{“isChatInterface?”:true,“positionChat”:“Chat Down Full”,“scaleChat”:false,“enablePushToTalk”:true,“conversationMode”:“PushToTalk”},“brand”:{“isLogoOnScreen?”:false,“positionLogoScreen”:“Logo UP Right”,“sizeLogoScreen”:0.5}})
index.js:2 ExpConfig: true {isChatInterface?: true, positionChat: ‘Chat Down Full’, scaleChat: false, enablePushToTalk: true, conversationMode: ‘PushToTalk’}
index.js:2 Received message from UE: ToggleWebCam({“enable”: false,“port”:“”})
index.js:2 DEBUG URL
index.js:2 Reconnect logs false false true Loading
index.js:2 DEBUG URL
index.js:2 Received message from UE: ExpConfig({“device”:“PC”,“camera”:{“fOV”:69.077880859375,“panUpDown”:3,“panRightLeft”:19,“zoom”:-62,“tilting”:0},“webCam”:{“visionInput”:false,“webcamPosition”:“WebCam Down Left”,“webcamSize”:0.75},“chat”:{“isChatInterface?”:true,“positionChat”:“Chat Down Full”,“scaleChat”:false,“enablePushToTalk”:true,“conversationMode”:“PushToTalk”},“brand”:{“isLogoOnScreen?”:false,“positionLogoScreen”:“Logo UP Left”,“sizeLogoScreen”:0.5}})
index.js:2 ExpConfig: true {isChatInterface?: true, positionChat: ‘Chat Down Full’, scaleChat: false, enablePushToTalk: true, conversationMode: ‘PushToTalk’}
index.js:2 Received message from UE: ToggleWebCam({“enable”: false,“port”:“”})
index.js:2 DEBUG URL
index.js:2 Original Resolution: 1920x1080
index.js:2 Scaled Resolution: 1920x1080
index.js:2 Received message from UE: ToggleWebCam({“enable”: false,“port”:“”})
index.js:2 Received message from UE: ExpConfig({“device”:“PC”,“camera”:{“fOV”:69.077880859375,“panUpDown”:3,“panRightLeft”:19,“zoom”:-62,“tilting”:0},“webCam”:{“visionInput”:false,“webcamPosition”:“WebCam Down Left”,“webcamSize”:0.75},“chat”:{“isChatInterface?”:true,“positionChat”:“Chat Down Full”,“scaleChat”:false,“enablePushToTalk”:true,“conversationMode”:“PushToTalk”},“brand”:{“isLogoOnScreen?”:false,“positionLogoScreen”:“Logo UP Left”,“sizeLogoScreen”:0.5}})
index.js:2 ExpConfig: true {isChatInterface?: true, positionChat: ‘Chat Down Full’, scaleChat: false, enablePushToTalk: true, conversationMode: ‘PushToTalk’}
index.js:2 DEBUG URL
index.js:2 Received message from UE: ToggleWebCam({“enable”: false,“port”:“”})
index.js:2 Received message from UE: ExpConfig({“device”:“PC”,“camera”:{“fOV”:69.077880859375,“panUpDown”:3,“panRightLeft”:19,“zoom”:-62,“tilting”:0},“webCam”:{“visionInput”:false,“webcamPosition”:“WebCam Down Left”,“webcamSize”:0.75},“chat”:{“isChatInterface?”:true,“positionChat”:“Chat Down Full”,“scaleChat”:false,“enablePushToTalk”:true,“conversationMode”:“PushToTalk”},“brand”:{“isLogoOnScreen?”:false,“positionLogoScreen”:“Logo UP Left”,“sizeLogoScreen”:0.5}})
index.js:2 ExpConfig: true {isChatInterface?: true, positionChat: ‘Chat Down Full’, scaleChat: false, enablePushToTalk: true, conversationMode: ‘PushToTalk’}
index.js:2 DEBUG URL
index.js:2 Received message from UE: ToggleMouseVisibility({“show”: true})
index.js:2 [Violation] Added non-passive event listener to a scroll-blocking ‘wheel’ event. Consider marking event handler as ‘passive’ to make the page more responsive. See
registerHoveringMouseEvents @ index.js:2
registerMouse @ index.js:2
setMouseInputEnabled @ index.js:2
(anonymous) @ index.js:2
set value @ index.js:2
set flag @ index.js:2
setFlagEnabled @ index.js:2
(anonymous) @ index.js:2
(anonymous) @ index.js:2
onResponse @ index.js:2
(anonymous) @ index.js:2
handleOnMessage @ index.js:2
sendrecvDataChannelController.handleOnMessage @ index.js:2
dataChannel.onmessage @ index.js:2
index.js:2 [Violation] Added non-passive event listener to a scroll-blocking ‘wheel’ event. Consider marking event handler as ‘passive’ to make the page more responsive. See
registerHoveringMouseEvents @ index.js:2
registerMouse @ index.js:2
setMouseInputEnabled @ index.js:2
(anonymous) @ index.js:2
(anonymous) @ index.js:2
onResponse @ index.js:2
(anonymous) @ index.js:2
handleOnMessage @ index.js:2
sendrecvDataChannelController.handleOnMessage @ index.js:2
dataChannel.onmessage @ index.js:2
index.js:2 Received message from UE: ToggleLoadingScreen({“show”: false})
index.js:2 DEBUG URL
index.js:2 Received message from UE: OnConnectionStateChanged({“server”: “Connected”,“avatar”:“Connected”})
index.js:2 Unhandled UE command: OnConnectionStateChanged({“server”: “Connected”,“avatar”:“Connected”})
index.js:2 Received message from UE: ToggleLoadingScreen({“show”: false})
index.js:2 Received message from UE: OnConnectionStateChanged({“server”: “Connected”,“avatar”:“Connected”})
index.js:2 Unhandled UE command: OnConnectionStateChanged({“server”: “Connected”,“avatar”:“Connected”})
index.js:2 Received message from UE: OnConnectionStateChanged({“server”: “Connected”,“avatar”:“Connected”})
index.js:2 Unhandled UE command: OnConnectionStateChanged({“server”: “Connected”,“avatar”:“Connected”})
index.js:2 Received message from UE: DisplayMessage({“name”: “Guest”,“message”:“”,“endofmessage”:true,“sentenceready”:true,“player”:true})
index.js:2 IncomingMessage Final
index.js:2 DEBUG URL
index.js:2 incomingMessage {name: ‘Guest’, message: ‘’, endofmessage: true, sentenceready: true, player: true}
index.js:2 Received message from UE: DisplayMessage({“name”: “the dream reader”,“message”:“Willkommen. Ich bin der Traumleser. Sag mir – was hast du in letzter Zeit geträum”,“endofmessage”:false,“sentenceready”:true,“player”:false})
index.js:2 IncomingMessage Final Willkommen. Ich bin der Traumleser. Sag mir – was hast du in letzter Zeit geträum
index.js:2 DEBUG URL
index.js:2 incomingMessage {name: ‘the dream reader’, message: ‘Willkommen. Ich bin der Traumleser. Sag mir – was hast du in letzter Zeit geträum’, endofmessage: false, sentenceready: true, player: false}
index.js:2 Received message from UE: DisplayMessage({“name”: “the dream reader”,“message”:“t? Ich sage dir, was es bedeutet.”,“endofmessage”:false,“sentenceready”:true,“player”:false})
index.js:2 IncomingMessage Final t? Ich sage dir, was es bedeutet.
index.js:2 Received message from UE: DisplayMessage({“name”: “the dream reader”,“message”:“”,“endofmessage”:true,“sentenceready”:true,“player”:false})
index.js:2 IncomingMessage Final
index.js:2 DEBUG URL
index.js:2 incomingMessage {name: ‘the dream reader’, message: ‘t? Ich sage dir, was es bedeutet.’, endofmessage: false, sentenceready: true, player: false}
index.js:2 DEBUG URL
index.js:2 incomingMessage {name: ‘the dream reader’, message: ‘’, endofmessage: true, sentenceready: true, player: false}
index.js:2 Received message from UE: DisplayMessage({“name”: “the dream reader”,“message”:“”,“endofmessage”:true,“sentenceready”:true,“player”:false})
index.js:2 IncomingMessage Final
index.js:2 DEBUG URL
index.js:2 incomingMessage {name: ‘the dream reader’, message: ‘’, endofmessage: true, sentenceready: true, player: false}

2

Hello,

Welcome to the Convai Developer Forum!

Could you please share the experience link?

3

Thanks, you can access it here

and the experience as a standalone here:

*Both don’t answer

4

For me, it works. Which account are you using? Maybe you don’t have enough interaction.

5

We’re using bestellungen@diekavallerie.de

For the embed, the account I am logged in shouldn’t matter though.

We are based in Germany and our client is also in Germany. We both experience this issue.

6

Could you please record a video (making sure the console logs are open) and share it on Google Drive?

7

Sure thing, here you go https://drive.google.com/file/d/1gcFpsJu8ax2Ro5Kjiuwe5RPDLbnZcKOk/view?usp=drive_link

*Also, this time it worked a couple of times at first and then stopped working. This is after 5 mins and a couple of reloads. Maybe you can replicate it by refreshing the page until it stops working

8

FYI: I had the page open idle just now and he started speaking again from the start. Did you restart the session on the backend?

9

Could you please share the Character ID?

10

Character ID: f3c59244-21ce-11f1-8205-42010a7be02c

Also, heres the experience ID if you need it: 1c6da995-51cc-4bfb-a2a5-f84e7b8b361d

11

Could you please switch the voice to Convai or GCP voices?

12

I changed to jinx for testing for now. GCP doesnt seem to be available in the german + english setup

It still doesn’t work reliably unfortunately

13

What exactly doesn’t work? I’ve tested it extensively, and it works without any issues.

14

I was still having the same issue as before as shown in the video. It’s more stable with the convai voices, but it doesn’t seem to solve the underlying issue. I got it to produce the error once on the second reload of the page out of some 30-ish reloads

15

It may be related to your internet connection. Please make sure you’re not using a VPN or similar applications. I’ve been testing, but I haven’t been able to reproduce the issue.

In the video, I can see that you’re receiving responses, but the character isn’t speaking. As a final step, could you try using the Realtime 1.5 LLM?

Also, please note that ElevenLabs voices are higher in quality and larger in size compared to other voices, so response times may be longer.

16

I’ll try, but I’ve noticed something else in the responses in general. Are you giving the LLMs instructions on the backend and expecting a structured output of some sort? Some messages sometimes contain metadata like “emotion: neutral” or the character will say something like “neutral is not an emotion preset, I will use the next closest thing, which is calm” or something like that.

The incomingMessage in the log sometimes looks weirdly cut off too. Maybe the sentence splitter has issues with german?

Also, this line in the logs is sometimes different for different sessions:
”Reconnect logs false false true Loading”

17

It seems to work for now, at least the audio part. Sometimes the face freezes (the body still moves though) and sometimes the audio volume increases, seemingly after the first sentence, but other than that, it seems stable. Did you find the cause or can we help in debugging this issue further, should it occur again?

18

Could you clarify what you mean by “the face freezes while the body still moves”? I’m not sure I fully understand the exact behavior.

As for the voice getting louder, if you are using an ElevenLabs voice, that may be related to the voice provider, since the voice is generated on ElevenLabs’ side.

To identify the cause, we need to be able to clearly understand and reproduce the issue. So far, I have not been able to reproduce it or see any related error message.

If it happens again, please open F12 → Console and share the debug URL from the logs with us.

19

Sure, the face sometimes freezes in it’s expression. The body still moves, meaning the stream isn’t frozen, it’s the character. Depending on your implementaion, there might be an issue with how expressions get inferred or retrieved on the backend. I don’t know if there is a streaming mode in the service you use for receiving expressions, but if you’re using nvidias audio2face via API in streaming mode for example, it’s possible the connection drops every now and then, so the face doesn’t receive updates.

Can you tell me if you send audio and video to the client separately? That way I could see if there is a sync-issue or dropped packets for one or the other on our end. *I ask because sometimes the audio comes through a bit offset and some chunks are be played twice sometimes. In this session, some chunks were doubled: Convai Authentication

The voice getting louder may be an elevenlabs issue, it’s a bit subtle.

20

We’ll be releasing an update tomorrow that includes improvements related to this.

Also, we are not using Audio2Face. We use NeuroSync, our own lip sync system.

For more details, you can also take a look at these videos:
https://youtu.be/4fMCKkrfyaA
https://youtu.be/n-UG3nmMeZQ