AI assistant interrupted during playback

What do you need help with?

Bug report

Installation method

Manual installation / GitHub

Issue area

Connection / WebRTC

Impact

Major - a core flow is broken

Unreal Engine version

5.8.0

Convai Unreal Engine Plugin version

4.0.0-beta.24

Project setup

Blueprint

Target platform

Unreal Editor

Where does it happen?

Custom project

Short summary

During playback of an already received response, the AI assistant is unexpectedly interrupted approximately halfway through the sentence.

What happened?

During playback of an already received response, the AI assistant is unexpectedly interrupted approximately halfway through the sentence, even though there is no new response or user input at that moment. At the same time, the On Started Talking and On Finished Talking events are triggered in the log. After that, the current speech stops, but the character does not begin saying anything new. Could you please check why these events are triggered during active speech and whether they may be related to the interruption?

Unreal logs or error messages

LogBlueprintUserMessages: [WBP_Main_UI_C_0] START TALKING
ConvaiSubsystemLog: :microphone: Active speaker changed: convai-bot
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“epoch”:1,“label”:“rtvi-ai”,“neurosync_turn_id”:4,“response_id”:“1a88b6ac-560c-4d49-a98e-4436a8575a9d:r6”,“sequence”:60,“type”:“bot-stopped-speaking”}
ConvaiChatbotComponentLog: UConvaiChatbotComponent Transcription received: | Character ID : 0dfaadf8-5432-11f1-8aa0-42010a7be02e | Session ID : -1
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“epoch”:1,“neurosync_turn_id”:4,“response_id”:“1a88b6ac-560c-4d49-a98e-4436a8575a9d:r6”,“sequence”:61,“type”:“bot-turn-completed”,“was_interrupted”:true}
ConvaiSubsystemLog: Warning: OnDataPacketReceived: Unknown packet type ‘bot-turn-completed’.
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“bt_code”:“”,“bt_constants”:“”,“narrative_section_id”:“d263c07c-55d0-11f1-b153-42010a7be02e”,“type”:“behavior-tree-response”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“label”:“rtvi-ai”,“type”:“user-started-speaking”}
ConvaiChatbotComponentLog: InterruptSpeech: Interrupting character | Character ID : 0dfaadf8-5432-11f1-8aa0-42010a7be02e | Session ID : -1
ConvaiChatbotComponentLog: onAudioFinished - IsConnectionTalking: false
ConvaiFaceSyncLog: ConvaiPauseLipSync: Pausing lipsync - TotalFrames: 20, FramesRemaining: 9, TotalDuration: 0.333, TimePlayed: 0.179, TimeRemaining: 0.154, TotalPausedDuration: 0.000
ConvaiPlayerLog: Started talking
LogBlueprintUserMessages: [WBP_Main_UI_C_0] FINISH TALKING
LogBlueprintUserMessages: [WBP_Main_UI_C_0] “ArmHomeSpeakReminder: HOME, Armed=true, starting timer”
LogBlueprintUserMessages: [WBP_Main_UI_C_0] ResetInactivityTimer called
LogBlueprintUserMessages: [WBP_Main_UI_C_0] ResetInactivityTimer ->120.0
ConvaiFaceSyncLog: ConvaiStopLipSync: Stopping lipsync - bIsPlaying: true, bIsPaused: true, TotalFrames: 20, FramesRemaining: 9, TotalDuration: 0.333, TimePlayed: 0.179, TimeRemaining: 0.154, TotalPausedDuration: 0.000
ConvaiChatbotComponentLog: Character interrupted by user speaking
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“label”:“rtvi-ai”,“type”:“user-stopped-speaking”}
ConvaiChatbotComponentLog: Character interrupt ended
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“event_type”:“trigger-message”,“extras”:{“actual_run_llm”:“true”,“bot_state”:“speaking”,“downgrade_reason”:null,“downgraded”:false,“has_speak_tag”:false,“llm_triggered”:t

Screenshots, recording, logs, or minimal repro

Log_UE_ConvAI_Receive_StartFinish_Events.txt (1.1 MB)

I checked the relevant documentation, resources, and similar forum posts before posting.

on

I confirm that I did not include API keys, tokens, passwords, secrets, private character data, private conversation logs, or customer data.

on

Are you using Hands Free mode?

Also, does the same issue occur when you test with Push to Talk mode?

No, I am not using Hands Free mode. I am currently using Push to Talk.

Could you please test it on Avatar Studio? Are you seeing the same issue there?

Turned out the server could return empty lines in the response stream, and those were causing the interruption. Once I added handling to skip empty lines, everything works fine. Thanks!

Thanks for the update. Glad to hear you were able to resolve it.

Could you clarify whether this issue happens with the plugin in its default state, or was it caused by custom changes made on your side?

Yes, it was caused by my custom changes. I had added a 12 seconds fallback timer for cases when the assistant doesnt respond at all based on another topic I posted on this forum. The trigger was firing on empty messages, but since streamed responses can naturally contain empty lines, it was interrupting the character mid-speech. Once I removed that condition, the issue was gone.