Unity: 2022.3.62f1
Plugin: “NPC AI Engine - Dialog, actions, voice and lipsync” - 3.3.0 (Convai Essentials Kit)
Reallusion Creative Character Studio (Trial): 5.01
Context: Integrating a Convai-powered NPC into an AR experience using Unity.
Background:
I’ve built several Convai NPCs in Unreal Engine, but I’m new to both Unity and Convai’s Unity SDK. Since Unity offers broader AR capabilities, I’m transitioning development to that platform.
Current Setup:
In my Unity scene, I have a character that activates the Convai dialogue box when approached. Pressing the T key allows me to speak to the NPC, who responds with lip-sync animation. This setup works, but it requires two separate actions: proximity to trigger the UI, and a keypress to initiate voice input.
Investigation
Searched for Unity specific solutions on here and nothing seemed to correlate between youtube video and forum entry (the problem seemed to be covered in the Unreal forums, which uses a different approach). Apparently there a C# StartConversation() or StartListening() function call I need to place but neither seems to work (or is findable). Please help a blind man under a time crunch!
Goal:
How can I streamline this interaction so that a single event—such as pressing one button or flipping a switch—both opens the dialogue box and activates voice input, allowing the NPC to immediately listen and respond? The event flow and how to intercept doesn’t seem obvious in the Unity plugin.
Any videos or tutorials that are relevant are much appreciated.
The coded solution should ideally allow the viewer to talk by turning on, by button press, a virtual headset mic, without the proximity requirement. It’s like having a hidden headset for both human user and NPC (location triggering isn’t needed). View the scenario like we’re a SWAT team, and I need the NPC to answer questions and take instructions without looking at them or being near them.
Activate the NPC’s conversational mode with a single trigger event—like flipping a headset switch—that turns it on and keeps it active until explicitly turned off
You can use the SetActiveConvaiNPC method in ConvaiNPCManager.cs.
Pass the NPC you want to activate as the first parameter, and set the second parameter to false, this way the NPC won’t be deactivated again based on proximity.
Keep in mind that ConvaiNPCManager.cs is mainly for demo purposes, so you can and should customize it for your own project.
I called SetActiveConvaiNPC(NPC, false), button name is "Inactive”, but the NPC doesn’t respond to vocal commands afterward.
Then I assign StartTalking to a separate button (called “Talk”). When I press that button, the avatar picks up a few words but misses the rest. If I press the Talk button again it picks up a few more. I press it again the Avatar then replies with text/audio multiple times to each text block that occurred during the consecutive button presses.
Is there an extra step I need to take—like a continuity mechanism—to ensure the avatar receives the full vocal input for the entire duration of the human speech? The button would toggle on for human speaking to capture all the words, then the human would toggle off the human talk to allow the NPC to talk back.
I’ve got something working, though it’s a bit kludgy. I set up four buttons on the display to represent different stages of the NPC’s listening/talking behavior:
The Activate button behaves as expected—it sets the Convai NPC’s active state to false, which prevents it from being deactivated based on proximity (as you described).
The Talk button seems to initiate listening, although there’s no method available. I found that calling StartTalking() inside OnPointerDown() works, which suggests the listening state persists while the button is held down.
The Stop Listening button appears to finalize the user’s spoken input and trigger the NPC’s response. Based on this, I’ve included the call to StopListening() into OnPointerUp() to better align with the press-and-hold interaction.
A better setup would be a single press to open the channel (allowing full user-to-NPC dialogue), and another press to close it—similar to modern military headset behavior.
If there’s a recommended pattern for this kind of interaction, I’d love to hear it. Otherwise, I’ll keep duct-taping the current observations into a working system.
Thank you for your help in identifying the initial method call and parameters needed.
You’re on the right track. For this interaction style, the key point is that StartListening needs to run while you want the mic open, then StopListening closes and finalizes the user utterance.
Here are two common patterns:
1) Push-to-talk (hold)
OnPointerDown → StartListening()
OnPointerUp → StopListening()
public void OnPointerDown() {
convaiNpc.StartListening();
}
public void OnPointerUp() {
convaiNpc.StopListening();
}
2) Tap-to-toggle (single button)
Option A — with a coroutine that keeps listening “open”:
bool isListening;
Coroutine listenCo;
public void OnTogglePressed() {
if (!isListening) {
isListening = true;
listenCo = StartCoroutine(ListenLoop());
} else {
isListening = false;
if (listenCo != null) StopCoroutine(listenCo);
convaiNpc.StopListening();
}
}
IEnumerator ListenLoop() {
while (isListening) {
convaiNpc.StartListening(); // keep mic open
yield return null; // per-frame
}
}
Option B
bool isListening;
public void Update() {
if (!isListening) {
convaiNpc.StartListening();
} else {
convaiNpc.StopListening();
}
}
public void OnTogglePressed() {
if (!isListening) {
isListening = true;
} else {
isListening = false;
}
}
