Hi, We are getting fatal crashes in our packaged game, seemingly from Convai GRPC.
For context, our project is a public exhibition, so it is now live and running all day open to the public. The crashes seem to happen a couple of times a day.
I think it is related to Convai GRPC, around cleanup of closed connect, or inability to connect with the GRPC server. May also be related to the Enter Idle issue which we have had previously, and is still persisitng
This is happening across multiple characters / computers, but it’s worth noting that in each case, the game is only interacting with one character, but restarting sessions quite often. Is it possible that convai is trying to access a grpc thread that it has already destroyed?
example errors:
[2025.01.17-01.47.29:256][929]ConvaiGRPCLog: Received Audio Chunk: 3.238004 secs | Character ID : e0ec8566-9736-11ef-9701-42010a7be016 | Session ID : 1c59d945f1fa55c4f5fc6c4af1201ae2
[2025.01.17-01.47.29:256][929]ConvaiGRPCLog: EmotionResponse: neutral 2
[2025.01.17-01.47.29:256][929]ConvaiAudioStreamerLog: Warning: PlayVoiceSynced: Added Audio Chunk - Audio Duration: 3.237483
[2025.01.17-01.47.29:256][929]ConvaiGRPCLog: Received Text I'm Joi, a Virtual Memory Archivist. : | Character ID : e0ec8566-9736-11ef-9701-42010a7be016 | Session ID : 1c59d945f1fa55c4f5fc6c4af1201ae2 | ReceivedFinalResponse : False
[2025.01.17-01.47.29:256][929]ConvaiAudioStreamerLog: PlayLipSyncWithPreGeneratedDataSynced: Detected New LipSync Chunk ChunkDuration: 3.237483 ChunkLipSyncFrameRate: 100.000000 FrameIndex:0 ChunkFrameCounter: 333 ExpectedFrameCount:323.748291 ChunkFrameCounter: 333
[2025.01.17-01.47.29:257][929]LogBlueprintUserMessages: [BP_Joi_C_2] Emotion state changed
[2025.01.17-01.47.29:257][929]LogBlueprintUserMessages: [BP_Joi_C_2] Joi (v5-17/12): I'm Joi, a Virtual Memory Archivist. ,, D:3.236984
[2025.01.17-01.47.29:258][929]ConvaiAudioStreamerLog: Play Available Audio and LipSync - Audio Duration: 3.237483 - Audio Chunks: 1 - LipSync Duration: 0.710000 - LipSync Chunks: 1 - Audio Chunks Remaining: 0 - LipSync Chunks Remaining: 0
[2025.01.17-01.47.29:412][933]ConvaiAudioStreamerLog: PlayLipSyncWithPreGeneratedDataSynced: Failed to detect New LipSync Chunk due to insufficent audio chunks NumAudioChunks: 0 NumLipSyncChunks: 0 FrameIndex:0 ChunkFrameCounter: 324 ExpectedFrameCount:323.748291 ChunkFrameCounter: 324
[2025.01.17-01.47.29:687][941]ConvaiGRPCLog: Received Audio Chunk: 2.338005 secs | Character ID : e0ec8566-9736-11ef-9701-42010a7be016 | Session ID : 1c59d945f1fa55c4f5fc6c4af1201ae2
[2025.01.17-01.47.29:687][941]ConvaiGRPCLog: EmotionResponse: neutral 2
[2025.01.17-01.47.29:687][941]ConvaiAudioStreamerLog: Warning: PlayVoiceSynced: Added Audio Chunk - Audio Duration: 2.337483
[2025.01.17-01.47.29:687][941]ConvaiGRPCLog: Received Text I'll be interviewing you today. : | Character ID : e0ec8566-9736-11ef-9701-42010a7be016 | Session ID : 1c59d945f1fa55c4f5fc6c4af1201ae2 | ReceivedFinalResponse : False
[2025.01.17-01.47.29:687][941]ConvaiAudioStreamerLog: PlayLipSyncWithPreGeneratedDataSynced: Detected New LipSync Chunk ChunkDuration: 2.337483 ChunkLipSyncFrameRate: 100.000000 FrameIndex:0 ChunkFrameCounter: 325 ExpectedFrameCount:233.748291 ChunkFrameCounter: 325
[2025.01.17-01.47.29:688][941]LogBlueprintUserMessages: [BP_Joi_C_2] Emotion state changed
[2025.01.17-01.47.29:688][941]LogBlueprintUserMessages: [BP_Joi_C_2] Joi (v5-17/12): I'll be interviewing you today. ,, D:2.336984
[2025.01.17-01.47.29:689][941]ConvaiAudioStreamerLog: Play Available Audio and LipSync - Audio Duration: 2.337483 - Audio Chunks: 1 - LipSync Duration: 0.710000 - LipSync Chunks: 1 - Audio Chunks Remaining: 0 - LipSync Chunks Remaining: 0
[2025.01.17-01.47.29:822][945]ConvaiAudioStreamerLog: PlayLipSyncWithPreGeneratedDataSynced: Failed to detect New LipSync Chunk due to insufficent audio chunks NumAudioChunks: 0 NumLipSyncChunks: 0 FrameIndex:0 ChunkFrameCounter: 234 ExpectedFrameCount:233.748291 ChunkFrameCounter: 234
[2025.01.17-01.47.30:091][952]ConvaiGRPCLog: Received Audio Chunk: 2.225510 secs | Character ID : e0ec8566-9736-11ef-9701-42010a7be016 | Session ID : 1c59d945f1fa55c4f5fc6c4af1201ae2
[2025.01.17-01.47.30:091][952]ConvaiGRPCLog: EmotionResponse: neutral 2
[2025.01.17-01.47.30:091][952]ConvaiAudioStreamerLog: Warning: PlayVoiceSynced: Added Audio Chunk - Audio Duration: 2.224989
[2025.01.17-01.47.30:091][952]ConvaiGRPCLog: Received Text Could you introduce yourself? : | Character ID : e0ec8566-9736-11ef-9701-42010a7be016 | Session ID : 1c59d945f1fa55c4f5fc6c4af1201ae2 | ReceivedFinalResponse : False
[2025.01.17-01.47.30:091][952]ConvaiAudioStreamerLog: PlayLipSyncWithPreGeneratedDataSynced: Detected New LipSync Chunk ChunkDuration: 2.224989 ChunkLipSyncFrameRate: 100.000000 FrameIndex:0 ChunkFrameCounter: 235 ExpectedFrameCount:222.498871 ChunkFrameCounter: 235
[2025.01.17-01.47.30:092][952]LogBlueprintUserMessages: [BP_Joi_C_2] Emotion state changed
[2025.01.17-01.47.30:092][952]LogBlueprintUserMessages: [BP_Joi_C_2] Joi (v5-17/12): Could you introduce yourself? ,, D:2.22449
[2025.01.17-01.47.30:092][952]ConvaiAudioStreamerLog: Play Available Audio and LipSync - Audio Duration: 2.224989 - Audio Chunks: 1 - LipSync Duration: 0.710000 - LipSync Chunks: 1 - Audio Chunks Remaining: 0 - LipSync Chunks Remaining: 0
[2025.01.17-01.47.30:252][956]ConvaiAudioStreamerLog: PlayLipSyncWithPreGeneratedDataSynced: Failed to detect New LipSync Chunk due to insufficent audio chunks NumAudioChunks: 0 NumLipSyncChunks: 0 FrameIndex:0 ChunkFrameCounter: 223 ExpectedFrameCount:222.498871 ChunkFrameCounter: 223
[2025.01.17-01.47.30:591][966]ConvaiGRPCLog: Received Audio Chunk: 5.425510 secs | Character ID : e0ec8566-9736-11ef-9701-42010a7be016 | Session ID : 1c59d945f1fa55c4f5fc6c4af1201ae2
[2025.01.17-01.47.30:591][966]ConvaiGRPCLog: EmotionResponse: neutral 2
[2025.01.17-01.47.30:591][966]LogBlueprintUserMessages: [BP_Joi_C_2] Emotion state changed
[2025.01.17-01.47.30:592][966]ConvaiAudioStreamerLog: Warning: PlayVoiceSynced: Added Audio Chunk - Audio Duration: 5.424989
[2025.01.17-01.47.30:592][966]ConvaiGRPCLog: Received Text And just out of curiosity, if you could have any superpower, what would it be and why?: | Character ID : e0ec8566-9736-11ef-9701-42010a7be016 | Session ID : 1c59d945f1fa55c4f5fc6c4af1201ae2 | ReceivedFinalResponse : False
[2025.01.17-01.47.30:592][966]ConvaiAudioStreamerLog: PlayLipSyncWithPreGeneratedDataSynced: Detected New LipSync Chunk ChunkDuration: 5.424989 ChunkLipSyncFrameRate: 100.000000 FrameIndex:0 ChunkFrameCounter: 224 ExpectedFrameCount:542.498901 ChunkFrameCounter: 224
[2025.01.17-01.47.30:592][966]LogBlueprintUserMessages: [BP_Joi_C_2] Joi (v5-17/12): And just out of curiosity, if you could have any superpower, what would it be and why?,, D:5.42449
[2025.01.17-01.47.30:593][966]ConvaiAudioStreamerLog: Play Available Audio and LipSync - Audio Duration: 5.424989 - Audio Chunks: 1 - LipSync Duration: 0.710000 - LipSync Chunks: 1 - Audio Chunks Remaining: 0 - LipSync Chunks Remaining: 0
[2025.01.17-01.47.30:703][969]ConvaiAudioStreamerLog: PlayLipSyncWithPreGeneratedDataSynced: Failed to detect New LipSync Chunk due to insufficent audio chunks NumAudioChunks: 0 NumLipSyncChunks: 0 FrameIndex:0 ChunkFrameCounter: 543 ExpectedFrameCount:542.498901 ChunkFrameCounter: 543
[2025.01.17-01.47.30:703][969]ConvaiGRPCLog: Chatbot Total Received Lipsync Responses: 1656 Responses
[2025.01.17-01.47.30:703][969]ConvaiChatbotComponentLog: Chatbot Total Received Audio: 16.534943 seconds
[2025.01.17-01.47.30:703][969]ConvaiGRPCLog: Received Text : | Character ID : e0ec8566-9736-11ef-9701-42010a7be016 | Session ID : 1c59d945f1fa55c4f5fc6c4af1201ae2 | ReceivedFinalResponse : True
[2025.01.17-01.47.30:705][969]LogBlueprintUserMessages: [BP_Joi_C_2] Joi (v5-17/12): ,Is Final, D:-inf
[2025.01.17-01.47.32:453][ 17]ConvaiPlayerLog: Started Talking
[2025.01.17-01.47.32:460][ 17]ConvaiAudioLog: Using Microphone (Shure MV7) as Audio capture device with NumChannels:1 and SampleRate:48000
[2025.01.17-01.47.32:461][ 17]ConvaiChatbotComponentLog: InterruptSpeech: Interrupting character | Character ID : e0ec8566-9736-11ef-9701-42010a7be016 | Session ID : 1c59d945f1fa55c4f5fc6c4af1201ae2
[2025.01.17-01.47.32:461][ 17]ConvaiChatbotComponentLog: UConvaiChatbotComponent Request Finished! | Character ID : e0ec8566-9736-11ef-9701-42010a7be016 | Session ID : 1c59d945f1fa55c4f5fc6c4af1201ae2
[2025.01.17-01.47.32:461][ 17]ConvaiGRPCLog: AsyncGetResponse started | Character ID : e0ec8566-9736-11ef-9701-42010a7be016 | Session ID : 1c59d945f1fa55c4f5fc6c4af1201ae2
[2025.01.17-01.47.32:461][ 17]ConvaiGRPCLog: GRPC GetResponse stream initialized | Character ID : e0ec8566-9736-11ef-9701-42010a7be016 | Session ID : 1c59d945f1fa55c4f5fc6c4af1201ae2
[2025.01.17-01.47.32:461][ 17]ConvaiGRPCLog: request: character_id: "e0ec8566-9736-11ef-9701-42010a7be016"
session_id: "1c59d945f1fa55c4f5fc6c4af1201ae2"
audio_config {
sample_rate_hertz: 16000
enable_facial_data: true
face_model: FACE_MODEL_OVR_MODEL_NAME
}
| Character ID : e0ec8566-9736-11ef-9701-42010a7be016 | Session ID : 1c59d945f1fa55c4f5fc6c4af1201ae2
[2025.01.17-01.47.32:461][ 17]ConvaiGRPCLog: Initial Stream Write | Character ID : e0ec8566-9736-11ef-9701-42010a7be016 | Session ID : 1c59d945f1fa55c4f5fc6c4af1201ae2
[2025.01.17-01.47.32:461][ 17]ConvaiGRPCLog: Initial Stream Read | Character ID : e0ec8566-9736-11ef-9701-42010a7be016 | Session ID : 1c59d945f1fa55c4f5fc6c4af1201ae2
[2025.01.17-01.47.32:461][ 17]LogBlueprintUserMessages: [BP_Joi_C_2] Interupted
[2025.01.17-01.47.32:672][ 23]ConvaiPlayerLog: FinishTalking calling FinishGetResponseStream
[2025.01.17-01.47.32:672][ 23]ConvaiGRPCLog: Finish Writing to audio data buffer
[2025.01.17-01.47.32:672][ 23]ConvaiGRPCLog: FinishWriting:: Informing On Data Received
[2025.01.17-01.47.32:672][ 23]ConvaiGRPCLog: Calling Stream WritesDone | LastWriteReceived : True | AudioBuffer.Num() : 0 | Character ID : e0ec8566-9736-11ef-9701-42010a7be016 | Session ID : 1c59d945f1fa55c4f5fc6c4af1201ae2
[2025.01.17-01.47.32:672][ 23]ConvaiGRPCLog: On Stream Write Done Writing
[2025.01.17-01.47.32:672][ 23]ConvaiPlayerLog: Finished Talking
[2025.01.17-01.47.32:672][ 23]ConvaiGRPCLog: OnStreamWriteDone
[2025.01.17-01.47.32:672][ 23]ConvaiGRPCLog: NumberOfAudioBytesSent 3410
[2025.01.17-01.47.32:932][ 30]ConvaiGRPCLog: GetResponse EmotionResponseDebug: session_id: "1c59d945f1fa55c4f5fc6c4af1201ae2"
emotion_response: "Ecstasy Admiration Anticipation Distraction Apprehension Pensiveness"
[2025.01.17-01.47.33:000][ 32]ConvaiGRPCLog: Narrative Section Received a12db43c-9bc7-11ef-85ad-42010a7be016: | Character ID : e0ec8566-9736-11ef-9701-42010a7be016 | Session ID : 1c59d945f1fa55c4f5fc6c4af1201ae2 | ReceivedFinalResponse : False
[2025.01.17-01.47.33:000][ 32]LogBlueprintUserMessages: [BP_Joi_C_2] Narative Section recieved: Welcome_UserEnter (a12db43c-9bc7-11ef-85ad-42010a7be016), Decisions: {2}
[2025.01.17-01.47.33:403][ 43]LogStreaming: Display: 0.001 ms for processing 66 objects in RemoveUnreachableObjects(Queued=0, Async=0). Removed 0 (845->845) packages and 0 (1539->1539) public exports.
[2025.01.17-01.47.33:439][ 44]ConvaiAudioStreamerLog: onAudioFinished
[2025.01.17-01.47.33:440][ 44]ConvaiGRPCLog: Destroying UConvaiGRPCGetResponseProxy... | Character ID : e0ec8566-9736-11ef-9701-42010a7be016 | Session ID : 1c59d945f1fa55c4f5fc6c4af1201ae2
[2025.01.17-01.47.33:440][ 44]ConvaiGRPCLog: Destroying UConvaiGRPCGetResponseProxy... | Character ID : e0ec8566-9736-11ef-9701-42010a7be016 | Session ID : 45ad90029067a5b07f6de62ea08df138
[2025.01.17-01.48.33:481][734]LogOutputDevice: Warning:
Script Stack (0 frames) :
[2025.01.17-01.48.33:481][734]LogStats: FPlatformStackWalk::StackWalkAndDump - 0.000 s
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: === Handled ensure: ===
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error:
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: Ensure condition failed: ReaderIndex != INDEX_NONE [File:D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\Misc\MTAccessDetector.h] [Line: 568]
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: Either a race detected (1 reader(s) on another thread(s) while acquiring write access on the current thread) or the access detector was trivially relocated:
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: Current thread 2916 callstack:
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6cdbc8d81 jm_mod_forever.exe!FMRSWRecursiveAccessDetector::GetCurrentThreadCallstack() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6db2113a6 jm_mod_forever.exe!`FMRSWRecursiveAccessDetector::AcquireWriteAccess'::`5'::<lambda_2>::operator()() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6cdbae8fa jm_mod_forever.exe!TDelegateAccessHandlerBase<FNotThreadSafeDelegateMode>::FWriteAccessScope::FWriteAccessScope() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6cdc3d9ca jm_mod_forever.exe!TDelegateBase<FNotThreadSafeDelegateMode>::~TDelegateBase<FNotThreadSafeDelegateMode>() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d802aa7b jm_mod_forever.exe!UConvaiGRPCGetResponseProxy::~UConvaiGRPCGetResponseProxy() [B:\perforce\jm-mod\jm-mod-forever\Plugins\Convai\Intermediate\Build\Win64\UnrealGame\Inc\Convai\UHT\ConvaiGRPC.gen.cpp:186]
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d802b8d4 jm_mod_forever.exe!UConvaiGRPCGetResponseProxy::`scalar deleting destructor'() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6ce6dfb74 jm_mod_forever.exe!FAsyncPurge::TickDestroyGameThreadObjects() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6ce678763 jm_mod_forever.exe!FProperty::ImportText_Internal() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6ce678da6 jm_mod_forever.exe!IncrementalPurgeGarbage() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d4d3d03f jm_mod_forever.exe!UEngine::ConditionalCollectGarbage() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d4213d49 jm_mod_forever.exe!UWorld::Tick() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d3f64139 jm_mod_forever.exe!UGameEngine::Tick() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d55c4d5a jm_mod_forever.exe!FEngineLoop::Tick() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d55dd0ac jm_mod_forever.exe!GuardedMain() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d55dd17a jm_mod_forever.exe!GuardedMainWrapper() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d55dfd45 jm_mod_forever.exe!LaunchWindowsStartup() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d55f0024 jm_mod_forever.exe!WinMain() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6dadbbe4a jm_mod_forever.exe!__scrt_common_main_seh() [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288]
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ffea155259d KERNEL32.DLL!UnknownFunction []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ffea1faaf38 ntdll.dll!UnknownFunction []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error:
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: Stack:
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6db2113ea jm_mod_forever.exe!`FMRSWRecursiveAccessDetector::AcquireWriteAccess'::`5'::<lambda_2>::operator()() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6cdbae8fa jm_mod_forever.exe!TDelegateAccessHandlerBase<FNotThreadSafeDelegateMode>::FWriteAccessScope::FWriteAccessScope() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6cdc3d9ca jm_mod_forever.exe!TDelegateBase<FNotThreadSafeDelegateMode>::~TDelegateBase<FNotThreadSafeDelegateMode>() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d802aa7b jm_mod_forever.exe!UConvaiGRPCGetResponseProxy::~UConvaiGRPCGetResponseProxy() [B:\perforce\jm-mod\jm-mod-forever\Plugins\Convai\Intermediate\Build\Win64\UnrealGame\Inc\Convai\UHT\ConvaiGRPC.gen.cpp:186]
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d802b8d4 jm_mod_forever.exe!UConvaiGRPCGetResponseProxy::`scalar deleting destructor'() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6ce6dfb74 jm_mod_forever.exe!FAsyncPurge::TickDestroyGameThreadObjects() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6ce678763 jm_mod_forever.exe!FProperty::ImportText_Internal() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6ce678da6 jm_mod_forever.exe!IncrementalPurgeGarbage() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d4d3d03f jm_mod_forever.exe!UEngine::ConditionalCollectGarbage() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d4213d49 jm_mod_forever.exe!UWorld::Tick() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d3f64139 jm_mod_forever.exe!UGameEngine::Tick() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d55c4d5a jm_mod_forever.exe!FEngineLoop::Tick() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d55dd0ac jm_mod_forever.exe!GuardedMain() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d55dd17a jm_mod_forever.exe!GuardedMainWrapper() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d55dfd45 jm_mod_forever.exe!LaunchWindowsStartup() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d55f0024 jm_mod_forever.exe!WinMain() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6dadbbe4a jm_mod_forever.exe!__scrt_common_main_seh() [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288]
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ffea155259d KERNEL32.DLL!UnknownFunction []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ffea1faaf38 ntdll.dll!UnknownFunction []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error:
[2025.01.17-01.48.33:481][734]LogStats: SubmitErrorReport - 0.000 s
[2025.01.17-01.48.33:481][734]LogStats: SendNewReport - 0.000 s
[2025.01.17-01.48.33:481][734]LogStats: FDebug::EnsureFailed - 0.001 s
[2025.01.17-01.48.33:481][734]LogOutputDevice: Warning:
Script Stack (0 frames) :
[2025.01.17-01.48.33:481][734]LogStats: FPlatformStackWalk::StackWalkAndDump - 0.000 s
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: === Handled ensure: ===
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error:
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: Ensure condition failed: LoadState().Value == ExpectedState.Value [File:D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\Misc\MTAccessDetector.h] [Line: 502]
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: Race detector destroyed while being accessed on another thread: 1 readers, 4095 writers on thread 4294967295:
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: Current thread 2916 callstack:
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6cdbc8d81 jm_mod_forever.exe!FMRSWRecursiveAccessDetector::GetCurrentThreadCallstack() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6db21109b jm_mod_forever.exe!`FMRSWRecursiveAccessDetector::~FMRSWRecursiveAccessDetector'::`2'::<lambda_1>::operator()() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6cdbafac6 jm_mod_forever.exe!TDelegateAccessHandlerBase<FNotThreadSafeDelegateMode>::~TDelegateAccessHandlerBase<FNotThreadSafeDelegateMode>() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d802aa7b jm_mod_forever.exe!UConvaiGRPCGetResponseProxy::~UConvaiGRPCGetResponseProxy() [B:\perforce\jm-mod\jm-mod-forever\Plugins\Convai\Intermediate\Build\Win64\UnrealGame\Inc\Convai\UHT\ConvaiGRPC.gen.cpp:186]
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d802b8d4 jm_mod_forever.exe!UConvaiGRPCGetResponseProxy::`scalar deleting destructor'() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6ce6dfb74 jm_mod_forever.exe!FAsyncPurge::TickDestroyGameThreadObjects() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6ce678763 jm_mod_forever.exe!FProperty::ImportText_Internal() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6ce678da6 jm_mod_forever.exe!IncrementalPurgeGarbage() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d4d3d03f jm_mod_forever.exe!UEngine::ConditionalCollectGarbage() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d4213d49 jm_mod_forever.exe!UWorld::Tick() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d3f64139 jm_mod_forever.exe!UGameEngine::Tick() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d55c4d5a jm_mod_forever.exe!FEngineLoop::Tick() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d55dd0ac jm_mod_forever.exe!GuardedMain() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d55dd17a jm_mod_forever.exe!GuardedMainWrapper() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d55dfd45 jm_mod_forever.exe!LaunchWindowsStartup() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d55f0024 jm_mod_forever.exe!WinMain() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6dadbbe4a jm_mod_forever.exe!__scrt_common_main_seh() [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288]
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ffea155259d KERNEL32.DLL!UnknownFunction []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ffea1faaf38 ntdll.dll!UnknownFunction []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error:
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: Writer thread 4294967295 callstack:
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [none]
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: Stack:
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6db21111b jm_mod_forever.exe!`FMRSWRecursiveAccessDetector::~FMRSWRecursiveAccessDetector'::`2'::<lambda_1>::operator()() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6cdbafac6 jm_mod_forever.exe!TDelegateAccessHandlerBase<FNotThreadSafeDelegateMode>::~TDelegateAccessHandlerBase<FNotThreadSafeDelegateMode>() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d802aa7b jm_mod_forever.exe!UConvaiGRPCGetResponseProxy::~UConvaiGRPCGetResponseProxy() [B:\perforce\jm-mod\jm-mod-forever\Plugins\Convai\Intermediate\Build\Win64\UnrealGame\Inc\Convai\UHT\ConvaiGRPC.gen.cpp:186]
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d802b8d4 jm_mod_forever.exe!UConvaiGRPCGetResponseProxy::`scalar deleting destructor'() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6ce6dfb74 jm_mod_forever.exe!FAsyncPurge::TickDestroyGameThreadObjects() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6ce678763 jm_mod_forever.exe!FProperty::ImportText_Internal() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6ce678da6 jm_mod_forever.exe!IncrementalPurgeGarbage() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d4d3d03f jm_mod_forever.exe!UEngine::ConditionalCollectGarbage() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d4213d49 jm_mod_forever.exe!UWorld::Tick() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d3f64139 jm_mod_forever.exe!UGameEngine::Tick() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d55c4d5a jm_mod_forever.exe!FEngineLoop::Tick() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d55dd0ac jm_mod_forever.exe!GuardedMain() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d55dd17a jm_mod_forever.exe!GuardedMainWrapper() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d55dfd45 jm_mod_forever.exe!LaunchWindowsStartup() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6d55f0024 jm_mod_forever.exe!WinMain() []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ff6dadbbe4a jm_mod_forever.exe!__scrt_common_main_seh() [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288]
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ffea155259d KERNEL32.DLL!UnknownFunction []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error: [Callstack] 0x00007ffea1faaf38 ntdll.dll!UnknownFunction []
[2025.01.17-01.48.33:481][734]LogOutputDevice: Error:
[2025.01.17-01.48.33:481][734]LogStats: SubmitErrorReport - 0.000 s
[2025.01.17-01.48.33:481][734]LogStats: SendNewReport - 0.000 s
[2025.01.17-01.48.33:481][734]LogStats: FDebug::EnsureFailed - 0.000 s
[2025.01.17-01.48.33:481][734]LogWindows: Could not start crash report client using ../../../Engine/Binaries/Win64/CrashReportClient-Win64-Debug.exe
[2025.01.17-01.48.33:481][734]LogMemory: Platform Memory Stats for Windows
[2025.01.17-01.48.33:481][734]LogMemory: Process Physical Memory: 2429.64 MB used, 2429.64 MB peak
[2025.01.17-01.48.33:481][734]LogMemory: Process Virtual Memory: 5454.25 MB used, 5624.57 MB peak
[2025.01.17-01.48.33:481][734]LogMemory: Physical Memory: 6668.11 MB used, 25862.33 MB free, 32530.44 MB total
[2025.01.17-01.48.33:481][734]LogMemory: Virtual Memory: 10602.55 MB used, 26791.89 MB free, 37394.44 MB total
[2025.01.17-01.48.33:481][734]LogBlueprintUserMessages: [BP_MOD_GameInstance_C_2147482554] JSON:new interaction
[2025.01.17-01.48.33:481][734]LogThreadingWindows: Error: Runnable thread gRPC_Stub crashed.
[2025.01.17-01.48.33:481][734]LogWindows: Error: === Critical error: ===
[2025.01.17-01.48.33:481][734]LogWindows: Error:
[2025.01.17-01.48.33:481][734]LogWindows: Error:
[2025.01.17-01.48.33:481][734]LogWindows: Error: Crash in runnable thread gRPC_Stub
[2025.01.17-01.48.33:497][735]LogExit: Executing StaticShutdownAfterError
[2025.01.17-01.48.33:498][735]LogWindows: FPlatformMisc::RequestExit(1, FRunnableThreadWin::GuardedRun.ExceptionHandler)
[2025.01.17-01.48.33:498][735]LogWindows: FPlatformMisc::RequestExitWithStatus(1, 3, FRunnableThreadWin::GuardedRun.ExceptionHandler)
[2025.01.17-01.48.33:498][735]LogCore: Engine exit requested (reason: Win RequestExit)
Multiple full crash logs can be found here:
As you can see the logs are very long, as the character is interacting all day.
Sometimes the crash logs are also not as verbose but i think the error is still related to closing the gRpc connection
[2025.01.18-01.12.51:604][500]ConvaiGRPCLog: No more data to read after 3 attempts. Calling Finish...
[2025.01.18-01.12.51:604][500]ConvaiGRPCLog: Calling Stream Finish | Character ID : 767979fc-9b2b-11ef-801e-42010a7be016 | Session ID : e17bc6bc03c005453f4b3832bc4fc48c
[2025.01.18-01.12.51:604][500]ConvaiGRPCLog: On Stream Finish | Character ID : 767979fc-9b2b-11ef-801e-42010a7be016 | Session ID : e17bc6bc03c005453f4b3832bc4fc48c
[2025.01.18-01.12.51:604][500]ConvaiChatbotComponentLog: UConvaiChatbotComponent Request Finished! | Character ID : 767979fc-9b2b-11ef-801e-42010a7be016 | Session ID : e17bc6bc03c005453f4b3832bc4fc48c
[2025.01.18-01.12.51:639][502]LogStreaming: Display: 0.001 ms for processing 2 objects in RemoveUnreachableObjects(Queued=0, Async=0). Removed 0 (841->841) packages and 0 (1529->1529) public exports.
[2025.01.18-01.12.51:658][503]ConvaiGRPCLog: Destroying UConvaiGRPCGetResponseProxy... | Character ID : 767979fc-9b2b-11ef-801e-42010a7be016 | Session ID : e17bc6bc03c005453f4b3832bc4fc48c
[2025.01.18-01.12.51:658][503]ConvaiGRPCLog: Destroying UConvaiGRPCGetResponseProxy... | Character ID : 767979fc-9b2b-11ef-801e-42010a7be016 | Session ID : e17bc6bc03c005453f4b3832bc4fc48c
[2025.01.18-01.13.51:682][504]LogWindows: Could not start crash report client using ../../../Engine/Binaries/Win64/CrashReportClient-Win64-Debug.exe
[2025.01.18-01.13.51:682][504]LogMemory: Platform Memory Stats for Windows
[2025.01.18-01.13.51:682][504]LogMemory: Process Physical Memory: 2140.95 MB used, 2291.89 MB peak
[2025.01.18-01.13.51:682][504]LogMemory: Process Virtual Memory: 5178.98 MB used, 5495.97 MB peak
[2025.01.18-01.13.51:682][504]LogMemory: Physical Memory: 6796.45 MB used, 25733.98 MB free, 32530.44 MB total
[2025.01.18-01.13.51:682][504]LogMemory: Virtual Memory: 11085.05 MB used, 23493.38 MB free, 34578.44 MB total
[2025.01.18-01.13.51:682][504]LogWindows: Error: === Critical error: ===
[2025.01.18-01.13.51:682][504]LogWindows: Error:
[2025.01.18-01.13.51:682][504]LogWindows: Error: Fatal error!
[2025.01.18-01.13.51:682][504]LogWindows: Error:
[2025.01.18-01.13.51:682][504]LogWindows: Error:
[2025.01.18-01.13.51:704][504]LogExit: Executing StaticShutdownAfterError
[2025.01.18-01.13.51:705][504]LogWindows: FPlatformMisc::RequestExit(1, LaunchWindowsStartup.ExceptionHandler)
[2025.01.18-01.13.51:705][504]LogWindows: FPlatformMisc::RequestExitWithStatus(1, 3, LaunchWindowsStartup.ExceptionHandler)
[2025.01.18-01.13.51:705][504]LogCore: Engine exit requested (reason: Win RequestExit)
We are using UE5.4.4 and Convai plugin 3.5.0
Any help is greatly appreciated, we’d love to get to the bottom of this issue quickly.