Conversation issues with a new character

Core Platform Voice, Speech & Language
1

What do you need help with?

Bug report

Issue area

Other

Impact

Blocker - I cannot continue

Character ID

929ec9c4-869e-11f1-b5c8-42010a7be02f and ad87dc5e-869d-11f1-90c3-42010a7be02f

Where does it happen?

Unity SDK

Language

English, Italian, French, Spanish, German, Japanese, Dutch, Chinese

Voice provider

Eleven Labs

Voice name

Nicole Enhanced

Environment

Meta Quest 3

Short summary

Character often doesn’t respond, when responding the response is broken up into smaller chunks of a few words with silence gaps in-between

User input

Please see the video.

i.e. “Hi, how are you?”

What happened?

Character often doesn’t respond, when responding the response is broken up into smaller chunks of a few words with silence gaps in-between, and my speech is often not transcribed into text.

I tested this character briefly in the Playground, it seemed ok.

I don’t think it’s my app though, other characters although do this sometimes they seem mostly ok. When I was the only person speaking to the character (when they weren’t released) these issues didn’t happen, it’s only since I released the character into my app in general did these issues occur (it seems).

There’s 2 character ID’s because this is the same character setup i.e. Core Description etc, just with different languages enabled, so more users can interact etc.

Here’s a vid showing the issues :

There are no error messages appearing.

This seems related to this possibly Is there an issue with Eleven Labs voices? (the status page doesn't show a problem)

What did you expect?

For my speech to be transcribed into text and for the character to respond smoothly.

Steps to reproduce

  1. Say “Hi how are you?”

I checked the relevant documentation, resources, and similar forum posts before posting.

on

I confirm that I did not include API keys, tokens, passwords, secrets, private character data, private conversation logs, or customer data.

on

2

Are the characters experiencing the issue using the same LLM and voice provider/voice as the characters that are working correctly?

Please compare the LLM and voice settings between the characters that have this issue and the ones that do not. This will help us narrow down whether the behavior is related to a specific model, voice, or character configuration.

3

I have spoken to one fully working character about 15 minutes ago, it was using the Gemma 4 31B LLM and the Ophelia eleven labs voice.

I have just changed the character to the Nicole Enhanced voice, so this is the same LLM and voice that cause problems with the other character that this thread is about, and there were similar issues.

I changed the voice on the problem character to that of a known working voice (Ophelia EL), the character still has the issue.

The LLM of the problem character has mostly been Gemma 4 31B and it worked fine for about a week, it became problematic so I tried Grok 4.3, still the same. Several of the characters us Gemma 4 31B ok.

There is one other character that uses the same LLM and voice and I’ve just seen it has the same issues. Non EL voices don’t seem to have this problem, the only consistent thing it seems is EL voices but I don’t see how that would affect my voice being transcribed.

EDIT : I’ve tested more characters with non EL voices and the issues happen with those too, so it’s not just EL voices. Let me know if you need any more info from me.

4

Hi K3, it seems the conversation issues I am experiencing was to do with being on an older SDK and specifically to do with the gRPC pipeline, so I used the SDK’s V4.4.0 REST endpoints and conversation worked great again.

My project has a load of custom things embedded in SDK 3.x Convai scripts so I am doing a partial SDK update for now just to get conversation working again, which I did last night, few things not working like animations but will fix them.

One thing I did notice with the character with the character ID I gave further up, they were quite a bit different in personality and response i.e. length of response, etc than they were with SDK 3.x. Just didn’t expect that, didn’t know the SDK version could affect how the character behaves, is that not correct?

The hands free is awesome, I provided the version I have partly working to my Patrons last night and several people commented how fast and responsive the conversation is. I did have VAD when using SDK 3.x but it was a hack and not as good as Convai’s version.

5

Convai Unity SDK v.4.4.0 is really good, why did I wait so long to update! not quite completed the update yet, I’m doing it step by step with just 1 of the 18 characters in my app but what is working is really good, my Patrons are really liking it (I provided them an early unfinished version). Well done Convai :+1: