Conversation breaks when user try to interrupt AIAssistant

Integrations Unreal Engine Plugin
1

What do you need help with?

Bug report

Installation method

Manual installation / GitHub

Issue area

Microphone / Voice Input

Impact

Blocker - I cannot continue

Unreal Engine version

5.8.2

Convai Unreal Engine Plugin version

4 0 0 beta 28

Project setup

Blueprint

Target platform

Unreal Editor

Where does it happen?

Custom project

Short summary

Once you interrupt AIAssistant when it speaks conversation completely break.

What happened?

A conversation with the AI assistant works normally until the user interrupts the assistant while it is speaking.

When I press the button to interrupt the assistant, the assistant stops speaking after approximately one second. However, during that short delay, the system appears to capture the assistant’s own voice and treats the last word spoken by the assistant as if it was spoken by me.

This word immediately appears in the conversation transcript as a user message.

When I then start speaking, my actual speech is captured as a separate message. In some cases, the speech gets split into multiple consecutive user messages, as if the system keeps switching between different speech segments or conversation turns.

As a result:

  • the assistant’s own last spoken word is incorrectly recognized as user input;
  • the user’s actual speech may be split into two or more separate messages;
  • the conversation flow becomes fragmented;
  • the AI assistant takes significantly longer to process the input;
  • the final response may sometimes be relevant, but in other cases it is unrelated or based on the incorrectly captured input.

Steps to reproduce

  1. Launch the project in Unreal Engine.
  2. Start a conversation with the AI assistant.
  3. Wait until the AI assistant starts speaking.
  4. While the assistant is still speaking, press the “Talk” button.
  5. Try to interrupt the assistant and start speaking immediately.
  6. Observe the conversation transcript and the assistant’s behavior after the interruption.

Screenshots, recording, logs, or minimal repro

ConvAI_LOG_Blocker_Issue.txt (64.9 KB)

I checked the relevant documentation, resources, and similar forum posts before posting.

on

I confirm that I did not include API keys, tokens, passwords, secrets, private character data, private conversation logs, or customer data.

on

2

If you are using speakers, your microphone may be picking up the character’s voice while the assistant is speaking. In that case, the system can mistakenly detect the assistant’s last spoken word as user input.

Are you testing this with headphones, or are you using external speakers?

3

Hi, yes we are using an external speaker (HECATE G100) with a HyperX QuadCast S microphone. However, we tested the echo cancellation separately in beta.29.1, and under normal conditions it works extremely well. Even with the speaker very close to the microphone at high volume, the character normally does not transcribe its own speech.

We are seeing a very similar issue on Unreal Engine 5.7.4 with Convai Unreal Plugin 4.0.0-beta.29.1.

Our setup:

  • Unreal Engine 5.7.4
  • Convai Unreal Plugin 4.0.0-beta.29.1
  • Native Convai client: 0.2.14.1+eb84d82
  • Blueprint project
  • Push To Talk enabled
  • External speaker: HECATE G100
  • Microphone: HyperX QuadCast S

We have reproduced the issue multiple times and found a very consistent trigger.

Push To Talk works normally if we wait until the assistant completely finishes speaking.

However, if we press PTT while the assistant is still speaking, interrupt the assistant, and immediately ask a new question, that new user turn is often ignored completely.

In the Unreal log, the microphone and STT are working correctly and the new utterance is recognized as partial transcription (final:false).

On PTT release, the client sends:

force-user-stopped-speaking

The server responds successfully:

Manual user turn stop queued

and then:

STT muted

But in the failing case we never receive:

final-user-transcription

As a result:

  • user-llm-text is never generated
  • the LLM never starts
  • no Action or MCP/tool call is executed
  • the character gives no response

We have reproduced this specifically when interrupting an active bot response.

We also receive this warning when the session starts:

Client version unknown. Compatibility issues may occur.

Could you please confirm whether this is a known PTT/interruption turn-finalization issue in beta.29.1 or native client 0.2.14.1?

Is there an updated native client, server-side fix, newer plugin build, or recommended workaround available?

We can provide full Unreal logs with exact failing turns if needed.

4

Additional findings after further testing:

We have now confirmed that interruption is NOT required to reproduce the PTT failure.

In one clean repro, the previous assistant response had fully completed before the next PTT press. The log shows:

bot-stopped-speaking

bot-turn-completed

was_interrupted:false

Only after the assistant had completely finished did we press PTT and ask the next question.

STT correctly recognized the complete utterance, but it remained final:false.

On PTT release:

Sent force-user-stopped-speaking

The server successfully returned:

Manual user turn stop queued

followed by:

STT muted

However, final-user-transcription never arrived, so there was no user-llm-text, no LLM start, and no response.

A late duplicate of the same complete transcription then arrived after STT was already muted, still as final:false.

We then reproduced an even clearer case with another utterance.

The same complete sentence was spoken twice in separate PTT attempts. Both times:

  • the full sentence was correctly recognized as final:false
  • PTT release sent force-user-stopped-speaking
  • the server returned Manual user turn stop queued successfully
  • STT muted returned successfully
  • but no final-user-transcription was generated

Late copies of the same final:false transcription continued to arrive after mute.

Then, on the NEXT PTT press, the stale transcription from the previous attempt suddenly became:

final:true

and only then did we receive:

final-user-transcription

user-llm-text

bot-llm-started

and the assistant responded normally.

So the common issue appears to be that the PTT user turn is sometimes not finalized on release, even though the complete speech has already been recognized.

The stale transcription appears to survive force-user-stopped-speaking and STT muted, and may only finalize during a later PTT hold.

This is reproducible on:

  • Unreal Engine 5.7.4
  • Convai Unreal Plugin 4.0.0-beta.29.1
  • Native client 0.2.14.1+eb84d82

Could you please check whether this is a known turn-finalization issue in the native client or backend, and whether an updated client/server build or workaround is available?

We have full Unreal logs for both clean reproductions and can provide them if useful.

5

Please share the logs.