Shared room with two human participants – only the creator's voice is transcribed

What do you need help with?

Bug report

Issue area

Microphone Input

Impact

Blocker - I cannot continue and have no workaround

Character ID

43b6ec76-d589-11f1-96b7-42010a7be02f

Session ID

628dfa51-dfb8-4e49-8a30-829c5f6f2b6d

Where does it happen?

Unity SDK

Language

English

Voice provider

N/A

Voice name

N/A

Environment

We are building a two-learner VR simulation (Unity SDK 4.4.1, Meta Quest 3 and Quest Pro, HandsFree / smart turn) where two people talk to one character in the

Short summary

Can only hear one person’s voice

What happened?

  • Headset 2 joins the LiveKit room and publishes its microphone track (it appears as a “User-…” participant).
  • Only headset 1’s speech is transcribed and answered. Headset 2’s speech is never transcribed.
  • When headset 2 creates the room itself, its voice is transcribed normally, so its microphone works.

What did you expect?

  1. Does a shared room support speech-to-text for more than one human participant? If so, what configuration is required (join mode, per-participant STT, turn-detection settings)?
  2. Can transcripts carry the speaker identity (speaker_id / participant_id) to the LLM, so the character knows which learner is talking?
  3. When we interrupt a reply before it is played (interrupt-bot), it still appears in the session history. Is there a way to keep unplayed replies out of the LLM context?

I checked the relevant documentation, resources, and similar forum posts before posting.

on

I confirm that I did not include API keys, tokens, passwords, secrets, private character data, private conversation logs, or customer data.

on

Follow-up with more detail and a proposed fix — could you confirm whether our understanding is right?

Setup (Unity SDK 4.4.1, Quest 3 + Quest Pro, one character shared by two learners):

  • Headset A creates the room: mode “create”, spawn_agent true, shared_session_key, max_num_participants = 2.
  • Headset B joins it: mode “join”, same room_name, spawn_agent false, its own end_user_id.
  • Both connect in HandsFree mode (server smart turn, default_stt_enabled = true in the connect request).

What happens:

  • Headset B joins the LiveKit room and publishes its microphone track (we can see the “User-…” participant and its audio track).
  • Only headset A’s speech is transcribed and answered. The chat history for session 628dfa51-dfb8-4e49-8a30-829c5f6f2b6d (character 43b6ec76-d589-11f1-96b7-42010a7be02f, 2026-11-20 21:58 UTC) has a single “user” stream; headset B only appears as faint crosstalk picked up by A’s microphone.
  • When headset B creates the room itself, its voice is transcribed normally, so its microphone and permissions work.
  • In push-to-talk mode, two learners worked: there the SDK sends “stt-toggle {muted:false}” from each headset on every press and “force-user-stopped-speaking” on release.

Our hypothesis:
In HandsFree mode, server STT for a participant that joins an existing room is not switched on. The default_stt_enabled flag in a “join” request may be ignored, because the room’s STT state comes from the creator. Push-to-talk works only because each press explicitly sends stt-toggle from that headset.

Questions:

  1. Is the hypothesis right? Does a joined participant need its own stt-toggle, or some other setting, for HandsFree STT?
  2. Does the server’s smart turn / VAD run per participant, so two learners talking in the same room are segmented and transcribed separately?
  3. Can each transcript carry the speaker’s identity (speaker_id / participant_id) into the LLM context, so the character knows which learner is speaking?
  4. Replies we interrupt before they are played (interrupt-bot) still appear in the session history. Is there a supported way to keep unplayed replies out of the LLM context, or to make the character listen without generating a reply to some user turns?

Thank you!

Thanks for the report.

I’ve shared this with the team that owns this part of Convai. We will update this thread as soon as we know more.