We’re developing an experience where the Convai character is running inside an Unreal Engine build, which is displayed on a TV. Inside this 3D environment, there’s a screen or surface positioned behind the character — and we want that screen to display different images depending on what the character says.
We’ve already structured a narrative design with keyword-based triggers to dynamically switch those images, but we haven’t found a way to access or extract the exact text or keywords being spoken by the character in real time.
Our main question is:
Is there a way (via API or other method) to access the text response generated by the character before or during the speech, so we can use that data to trigger visual events inside Unreal?
Is there any existing feature or best practice to send custom events to Unreal Engine based on Convai’s narrative flow?
We appreciate any guidance you can provide on this — it would really help us move forward with this integration.
To clarify and guide you better, will the image changes be triggered by what the user says, or what the Convai character says? Since Narrative Design progresses based on the user’s input, understanding this flow is key to offering the most effective solution.
If you can share more details on how and when you’d like these triggers to happen, we’d be happy to help you set up the right approach!
Yes, we’d like to work with two types of triggers: one based on what the user says, and another based on what the character says during the narrative.
Triggers based on the character’s speech:
During a guided narrative (like a virtual tour), the character mentions points of interest. For example: “our lake is perfect for relaxing with family.” When the character says “lake,” we want a specific image to appear in the 3D scene (behind the character).
In other words, we’d like to trigger visual changes based on keywords spoken by the character, in sync with the narration.
Triggers based on the user’s speech:
If the user says something like “I want to see project X,” the character responds accordingly, and we want the image of project X to appear at that moment.
In this case, the user’s input should trigger a visual logic along with the character’s verbal response.
We’re already integrated with Unreal Engine, and structuring the flows using the Narrative Design API, including states, intents, and triggers.
We’d appreciate guidance on:
Whether there’s a recommended way to detect specific words in the character’s responses or to configure custom events that can be used to trigger visual changes?
For user inputs, can we set up intents or custom triggers to connect specific keywords to actions within the 3D scene?
Thanks for the detailed explanation. It’s really helpful to understand what you’re aiming for.
For triggers based on character responses:
You can use the OnTextReceived event from the ConvaiChatBot component in Unreal. This event lets you access the character’s response text. Once you have that, simply use a Contains check in your script or blueprint to see if the message includes a keyword like “lake.” Please note that this check is case-sensitive, so make sure to handle string casing accordingly. If the condition matches, you can then trigger any visual update in your scene.
For triggers based on user input:
This should be structured through Narrative Design. Each input from the user can lead to specific narrative sections by configuring Decisions clearly in your design. To ensure your flow works as intended:
Test the flow thoroughly in the Convai Playground.
Make sure your Decision inputs are explicit and unambiguous.
Once verified, use the OnNarrativeSectionReceived event (see this guide) to handle section transitions and trigger specific actions or visual updates based on the section ID.