Blocker - I cannot continue and have no workaround
Character ID
43b6ec76-d589-11f1-96b7-42010a7be02f
Session ID
628dfa51-dfb8-4e49-8a30-829c5f6f2b6d
Where does it happen?
Unity SDK
Language
English
Voice provider
N/A
Voice name
N/A
Environment
We are building a two-learner VR simulation (Unity SDK 4.4.1, Meta Quest 3 and Quest Pro, HandsFree / smart turn) where two people talk to one character in the
Short summary
Can only hear one person’s voice
What happened?
Headset 2 joins the LiveKit room and publishes its microphone track (it appears as a “User-…” participant).
Only headset 1’s speech is transcribed and answered. Headset 2’s speech is never transcribed.
When headset 2 creates the room itself, its voice is transcribed normally, so its microphone works.
What did you expect?
Does a shared room support speech-to-text for more than one human participant? If so, what configuration is required (join mode, per-participant STT, turn-detection settings)?
Can transcripts carry the speaker identity (speaker_id / participant_id) to the LLM, so the character knows which learner is talking?
When we interrupt a reply before it is played (interrupt-bot), it still appears in the session history. Is there a way to keep unplayed replies out of the LLM context?
I checked the relevant documentation, resources, and similar forum posts before posting.
on
I confirm that I did not include API keys, tokens, passwords, secrets, private character data, private conversation logs, or customer data.
Follow-up with more detail and a proposed fix — could you confirm whether our understanding is right?
Setup (Unity SDK 4.4.1, Quest 3 + Quest Pro, one character shared by two learners):
Headset A creates the room: mode “create”, spawn_agent true, shared_session_key, max_num_participants = 2.
Headset B joins it: mode “join”, same room_name, spawn_agent false, its own end_user_id.
Both connect in HandsFree mode (server smart turn, default_stt_enabled = true in the connect request).
What happens:
Headset B joins the LiveKit room and publishes its microphone track (we can see the “User-…” participant and its audio track).
Only headset A’s speech is transcribed and answered. The chat history for session 628dfa51-dfb8-4e49-8a30-829c5f6f2b6d (character 43b6ec76-d589-11f1-96b7-42010a7be02f, 2026-11-20 21:58 UTC) has a single “user” stream; headset B only appears as faint crosstalk picked up by A’s microphone.
When headset B creates the room itself, its voice is transcribed normally, so its microphone and permissions work.
In push-to-talk mode, two learners worked: there the SDK sends “stt-toggle {muted:false}” from each headset on every press and “force-user-stopped-speaking” on release.
Our hypothesis:
In HandsFree mode, server STT for a participant that joins an existing room is not switched on. The default_stt_enabled flag in a “join” request may be ignored, because the room’s STT state comes from the creator. Push-to-talk works only because each press explicitly sends stt-toggle from that headset.
Questions:
Is the hypothesis right? Does a joined participant need its own stt-toggle, or some other setting, for HandsFree STT?
Does the server’s smart turn / VAD run per participant, so two learners talking in the same room are segmented and transcribed separately?
Can each transcript carry the speaker’s identity (speaker_id / participant_id) into the LLM context, so the character knows which learner is speaking?
Replies we interrupt before they are played (interrupt-bot) still appear in the session history. Is there a supported way to keep unplayed replies out of the LLM context, or to make the character listen without generating a reply to some user turns?