The character occasionally fails to respond

Core Features Conversation Issues
1

Hi :slight_smile:

For a good while the characters in my Unity app for Quest 3 VR headsets have been failing to respond to the user now and again, at a very rough guess I’d say the character fails to respond once in about 12-15 responses but it varies. I’ve ignored it up to now but decided to look at it more closely today to try to eradicate it because it is off-putting.

I enabled all the Convai Logger Settings in Unity and noticed this output to my log file when the app runs, when the failed response happens :

10:42:33.992 <color=grey>[Info][Character]: Hannah is now listening</color>
[Stack Trace - Method: RunInternal, at Line: 0 in File: ]
10:42:36.970 <color=grey>[Info][Character]: Microphone is working fine.</color>
[Stack Trace - Method: InvokeMoveNext, at Line: 0 in File: ]
10:42:46.047 RpcException: Status(StatusCode=Internal, Detail="Get Response Failed None Stream removed")
10:42:46.047 Convai.Scripts.Runtime.Core.ConvaiGRPCAPI.ProcessAudioChunk (Grpc.Core.AsyncDuplexStreamingCall`2[TRequest,TResponse] call, System.Int32 diff, System.Collections.Generic.IReadOnlyList`1[T] audioData) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine] (TStateMachine& stateMachine) (at <00000000000000000000000000000000>:0)
Convai.Scripts.Runtime.Core.ConvaiGRPCAPI.ProcessAudioChunk (Grpc.Core.AsyncDuplexStreamingCall`2[TRequest,TResponse] call, System.Int32 diff, System.Collections.Generic.IReadOnlyList`1[T] audioData) (at <00000000000000000000000000000000>:0)
Convai.Scripts.Runtime.Core.ConvaiGRPCAPI.ProcessAudioContinuously (Grpc.Core.AsyncDuplexStreamingCall`2[TRequest,TResponse] call, System.Int32 recordingFrequency, System.Int32 recordingLength, UnityEngine.AudioClip audioClip) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
UnityEngine.UnitySynchronizationContext+WorkRequest.Invoke () (at <00000000000000000000000000000000>:0)
UnityEngine.UnitySynchronizationContext.Exec () (at <00000000000000000000000000000000>:0)
--- End of stack trace from previous location where exception was thrown ---
Convai.Scripts.Runtime.Core.ConvaiGRPCAPI.ProcessAudioContinuously (Grpc.Core.AsyncDuplexStreamingCall`2[TRequest,TResponse] call, System.Int32 recordingFrequency, System.Int32 recordingLength, UnityEngine.AudioClip audioClip) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
UnityEngine.UnitySynchronizationContext+WorkRequest.Invoke () (at <00000000000000000000000000000000>:0)
UnityEngine.UnitySynchronizationContext.Exec () (at <00000000000000000000000000000000>:0)
--- End of stack trace from previous location where exception was thrown ---
Convai.Scripts.Runtime.Core.ConvaiGRPCAPI.StartRecordAudio (Service.ConvaiService+ConvaiServiceClient client, System.Boolean isActionActive, System.Boolean isLipSyncActive, System.Int32 recordingFrequency, System.Int32 recordingLength, System.String characterID, Service.ActionConfig actionConfig, Service.FaceModel faceModel, System.String speakerID) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.AwaitTaskContinuation.RunCallback (System.Threading.ContextCallback callback, System.Object state, System.Threading.Tasks.Task& currentTask) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.FinishContinuations () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.Finish (System.Boolean bUserDelegateExecuted) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.TrySetException (System.Object exceptionObject) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetException (System.Exception exception) (at <00000000000000000000000000000000>:0)
Convai.Scripts.Runtime.Core.ConvaiGRPCAPI.ProcessAudioContinuously (Grpc.Core.AsyncDuplexStreamingCall`2[TRequest,TResponse] call, System.Int32 recordingFrequency, System.Int32 recordingLength, UnityEngine.AudioClip audioClip) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
UnityEngine.UnitySynchronizationContext+WorkRequest.Invoke () (at <00000000000000000000000000000000>:0)
UnityEngine.UnitySynchronizationContext.Exec () (at <00000000000000000000000000000000>:0)
--- End of stack trace from previous location where exception was thrown ---
Convai.Scripts.Runtime.Core.ConvaiNPC.StartListening () (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.AwaitTaskContinuation.RunCallback (System.Threading.ContextCallback callback, System.Object state, System.Threading.Tasks.Task& currentTask) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.FinishContinuations () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.Finish (System.Boolean bUserDelegateExecuted) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.TrySetException (System.Object exceptionObject) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetException (System.Exception exception) (at <00000000000000000000000000000000>:0)
Convai.Scripts.Runtime.Core.ConvaiGRPCAPI.StartRecordAudio (Service.ConvaiService+ConvaiServiceClient client, System.Boolean isActionActive, System.Boolean isLipSyncActive, System.Int32 recordingFrequency, System.Int32 recordingLength, System.String characterID, Service.ActionConfig actionConfig, Service.FaceModel faceModel, System.String speakerID) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.AwaitTaskContinuation.RunCallback (System.Threading.ContextCallback callback, System.Object state, System.Threading.Tasks.Task& currentTask) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.FinishContinuations () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.Finish (System.Boolean bUserDelegateExecuted) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.TrySetException (System.Object exceptionObject) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetException (System.Exception exception) (at <00000000000000000000000000000000>:0)
Convai.Scripts.Runtime.Core.ConvaiGRPCAPI.ProcessAudioContinuously (Grpc.Core.AsyncDuplexStreamingCall`2[TRequest,TResponse] call, System.Int32 recordingFrequency, System.Int32 recordingLength, UnityEngine.AudioClip audioClip) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
UnityEngine.UnitySynchronizationContext+WorkRequest.Invoke () (at <00000000000000000000000000000000>:0)
UnityEngine.UnitySynchronizationContext.Exec () (at <00000000000000000000000000000000>:0)
--- End of stack trace from previous location where exception was thrown ---
System.Runtime.CompilerServices.AsyncMethodBuilderCore+<>c.<ThrowAsync>b__7_0 (System.Object state) (at <00000000000000000000000000000000>:0)
UnityEngine.UnitySynchronizationContext+WorkRequest.Invoke () (at <00000000000000000000000000000000>:0)
UnityEngine.UnitySynchronizationContext.Exec () (at <00000000000000000000000000000000>:0)

10:42:48.385 <color=grey>[Info][Character]: Hannah has stopped listening</color>
[Stack Trace - Method: StopListening, at Line: 0 in File: ]

(Sorry for the wall of text)

This character is 12c5d804-bddd-11ef-905b-42010a7be016 and the failed responses happen with all characters.

2

Hello @Tyke,

Thanks for the details!

Does it also happen in the playground? If so, could you please share the session ID?

3

Hi K3, I’ve just chatted in the Playground just now and it didn’t happen. Not sure what to do about it to be honest.

4

So it only happens on the Unity side?

5

I asked the avatar about 15 questions in the playground and they replied every time. So yes it appears to only happen in Unity. Forum member Bob Hawkey’s Unity app says his has the same thing happening with occasional failed responses in fact I’ve just seen it happen today or yesterday in 1 of his video’s. I’m not sure if the same console error is generated but the way it failed is the same as in my app.

6

Hello @Tyke,

This is a known issue. We recommend waiting for our upcoming new SDK release.

1 Like
7

Sorry to hijack the topic, we’re having the same issue as reported by Tyke. It occurs randomly, making our user data collection process difficult.

Are there any known workarounds other than handling the exception client-side?

8

Could you please share the logs?

9

This is from our development build log. Today is the first time this has happened (and it happened several times consequently). This happened during long conversation sessions (~20 minutes).

Keep in mind we had to customize the Convai Unity client to support our specific use case (multi-scene setup, PCVR target, and client-side conversation logging).

This is pretty off-putting for the Player. They see the transcript in real time in the chatui but never receive an answer from the character.

(Filename: C:/Users/USER/Documents/Unity/counseling/Assets/Convai/Scripts/Runtime/LoggerSystem/ConvaiLogger.cs Line: 135)

<color=yellow>[Warning][Character]: RPC error during audio chunk processing: Internal - Status(StatusCode=Internal, Detail="Get Response Failed None Stream removed")</color>
[Stack Trace - Method: MoveNext, at Line: 926 in File: C:\Users\USER\Documents\Unity\counseling\Assets\Convai\Scripts\Runtime\Core\ConvaiGRPCAPI.cs]
UnityEngine.StackTraceUtility:ExtractStackTrace () (at C:/build/output/unity/unity/Runtime/Export/Scripting/StackTrace.cs:37)
UnityEngine.DebugLogHandler:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[])
UnityEngine.Logger:Log (UnityEngine.LogType,object)
UnityEngine.Debug:LogWarning (object)
Convai.Scripts.Runtime.LoggerSystem.ConvaiLogger:LogMessage (string,Convai.Scripts.Runtime.LoggerSystem.ConvaiLogger/LogLevel,Convai.Scripts.Runtime.LoggerSystem.ConvaiLogger/LogCategory,object[]) (at C:/Users/USER/Documents/Unity/counseling/Assets/Convai/Scripts/Runtime/LoggerSystem/ConvaiLogger.cs:90)
Convai.Scripts.Runtime.LoggerSystem.ConvaiLogger:Warn (string,Convai.Scripts.Runtime.LoggerSystem.ConvaiLogger/LogCategory,object[]) (at C:/Users/USER/Documents/Unity/counseling/Assets/Convai/Scripts/Runtime/LoggerSystem/ConvaiLogger.cs:135)
Convai.Scripts.Runtime.Core.ConvaiGRPCAPI/<ProcessAudioChunk>d__47:MoveNext () (at C:/Users/USER/Documents/Unity/counseling/Assets/Convai/Scripts/Runtime/Core/ConvaiGRPCAPI.cs:926)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start<Convai.Scripts.Runtime.Core.ConvaiGRPCAPI/<ProcessAudioChunk>d__47> (Convai.Scripts.Runtime.Core.ConvaiGRPCAPI/<ProcessAudioChunk>d__47&)
Convai.Scripts.Runtime.Core.ConvaiGRPCAPI:ProcessAudioChunk (Grpc.Core.AsyncDuplexStreamingCall`2<Service.GetResponseRequest, Service.GetResponseResponse>,int,single[])
Convai.Scripts.Runtime.Core.ConvaiGRPCAPI/<ProcessAudioContinuously>d__45:MoveNext () (at C:/Users/USER/Documents/Unity/counseling/Assets/Convai/Scripts/Runtime/Core/ConvaiGRPCAPI.cs:771)
System.Runtime.CompilerServices.AsyncMethodBuilderCore/MoveNextRunner:InvokeMoveNext (object)
System.Threading.ExecutionContext:RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool)
System.Threading.ExecutionContext:Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool)
System.Runtime.CompilerServices.AsyncMethodBuilderCore/MoveNextRunner:Run ()
System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation/<>c:<.cctor>b__7_0 (object)
UnityEngine.UnitySynchronizationContext/WorkRequest:Invoke () (at C:/build/output/unity/unity/Runtime/Export/Scripting/UnitySynchronizationContext.cs:153)
UnityEngine.UnitySynchronizationContext:Exec () (at C:/build/output/unity/unity/Runtime/Export/Scripting/UnitySynchronizationContext.cs:83)
UnityEngine.UnitySynchronizationContext:ExecuteTasks () (at C:/build/output/unity/unity/Runtime/Export/Scripting/UnitySynchronizationContext.cs:107)
10

Could you please share the Session ID?

11

This is the session (ID: c4a917cf0a14490509a6e27d9ef39e76) and the character I was talking about (ID: c3055c1c-0b0d-11f0-b5e8-42010a7be01a).

Thanks

12

@Tyke @massimo.zancanaro Do you keep the microphone on for a long time?

13

Yes, the player who was running this session recorded very long messages, using the full 30s available, during the last part of their playthrough

14

Not always, it can happen if the mic is only on for a few seconds, but it is more likely to fail if the mic is on for more than a few seconds, but it still sometimes happens anyway.

15

Yes, that’s probably where the error comes from.

16

Thanks for the quick reply.
So… Is this a known issue? Cause 30s for our use case is already pretty restrictive (counseling simulation scenario)

17

I did some tests on the message length. While this does seem to happen with long messages, it also occurs randomly in long conversation, as tyke said (here I’m talking message count-wise). And we’ve only just noticed this now, because, as I said, we’re currently running user tests, and our conversations typically last between 10 and 20 minutes.

Can you confirm whether this is a known issue? And if this has to do specifically with the Unity client?

Thank You

18

Hello, we’re still experiencing this issue, which is popping up randomly. At this point, we’ll likely implement a client-side solution to notify the player that the request has failed.

Again, can you confirm whether this is a known client-side issue specific to Unity or a server-side issue? Just to know if we should look out for future plugin updates?

Thank You

19

Hi Massimo, if you choose to add a client side message to indicate to the user the reply has failed can you briefly share with me please what exactly are you detecting, error wise? I’m thinking of doing this too, if I do it before you do it I will share what I’ve found.

20

The easiest way to implement something like this is to send a custom notification using the ConvAI notification system. We’re already doing this for API requests that fail with StatusCode=“Unavailable”.

The main idea is to wrap the specific statement throwing this exception (probably in ConvaiGRPCAPI.cs, see the error stack) in a try-catch block. Catch that exception/error and call the notification system in that specific catch block.

Example:

try
{
    // statements that throw that specific error/warning
}
catch (RpcException rpcException) when (rpcException.StatusCode == StatusCode.Unavailable)
{
    ConvaiLogger.DebugLog("Server unavailable during streaming", ConvaiLogger.LogCategory.Character);
    if (NotificationSystemHandler.Instance != null)
    {
        MainThreadDispatcher.Instance.RunOnMainThread(() => {
            try
            {
                NotificationSystemHandler.Instance.NotificationRequest(NotificationType.UserRequestFailed);
            }
            catch (System.Exception ex)
            {
                Debug.LogError($"Failed to create UI message: {ex.Message}");
            }
        });
    }
    OnUserRequestFailed?.Invoke();
    break; // Exit the loop when server is unavailable
}

Note: OnUserRequestFailed is a custom public event we defined. The notification instancing is wrapped in MainThreadDispatcher.Instance.RunOnMainThread, cause UI update operations/GameObject instancing operations need to run on the main thread.

1 Like