Follow me (Player) not working in VR (UE5.5.4 convai 4.0.0-beta.19)

[Nicolas_Leurquin]

Hello,

I’ve made a blank project from the VRTemplate and done the tutorials for setting up a metahuman and the VRPawn:
Real-Time AI Conversations & Facial Animation for MetaHumans | Convai Unreal Engine Tutorial

Add Conversational AI to MetaHuman, Reallusion and more | Convai UE Plugin Quick Setup Guide

And the one about enabling actions:
From Dialogue to Action: Create Scene-Aware AI Characters That Act AND React | Convai Unreal Engine
However, the NPC only moves towards the VRPawn origin, not the current camera location. It does seem to know where I am though or at least our relative orientations as it manages to turn towards me properly when I talk to it. If I ask the NPC to follow me again after it reached the VRPawn origin, it says:

“I’m already following you, so you’re all set. Just lead the way, and I’ll stay right with you. What’s next on our task list?”
Or:

“I’m right next to you now. Let me know what you’d like to do next or if there’s anything else I can assist with.”

, however she is standing still at VR Pawn origin

Could you look into this please? Thank you

[Nicolas_Leurquin]

In active level editor made, when i type “please follow me” she answers: { “actions”: [ { “name”: “Follow”, “target”: “User” } ] }, she literally says each word (not the brackets). I’ve seen another post mentioning this, i will read into it as well if anything is mentioned there

[Nicolas_Leurquin]

I tried again and move to me and follow me did work now with typing them in chat in active level editor viewport:

[Nicolas_Leurquin]

I take back the part about the NPC knowing where I am and turning their head towards me, I’m unsure whether it only worked once or if it was just turning to the VRPawn origin point (not the camera location)

[Nicolas_Leurquin]

I didn’t change anything to the project and now the NPC seems to turn their head towards me again, yet still doesn’t know my position for the follow me command, as in the original statement of the question.

[Nicolas_Leurquin]

I cannot find the code right away for setting the main character or other naming you have given it in this version, however from this blueprint I imagine it is the same issue as the previous version I tested: the actor location is being checked for distance with the NPC instead of the camera component. I could be wrong, I’m basically a UE beginner, so i’m a bit in the dark here if it is a problem that I can solve myself or that i require your help for

[K3]

Let’s go step by step.

Could you please test this on the First Person Level first? Please do not integrate any XR-related setup yet, and do not modify any plugin blueprints for this test.

This will help us confirm whether the base setup is working correctly before adding XR-specific elements.

[Nicolas_Leurquin]

Alright, I will try that

[Nicolas_Leurquin]

From my first attempt, she didn’t follow me when asked (typed through chat). Second attempt didn’t make her follow me either. She says: “Sure, I can do that. i’ll follow your lead. let me get right on it.”

[Nicolas_Leurquin]

ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“label”:“rtvi-ai”,“type”:“bot-llm-started”}
ConvaiFaceSyncLog: ConvaiStopLipSync: Stopping lipsync - bIsPlaying: false, bIsPaused: false, TotalFrames: 0, FramesRemaining: 0, TotalDuration: 0.000, TimePlayed: 0.000, TimeRemaining: 0.000, TotalPausedDuration: 0.000
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“label”:“rtvi-ai”,“type”:“user-started-speaking”}
ConvaiPlayerLog: Started talking
ConvaiFaceSyncLog: ConvaiStopLipSync: Stopping lipsync - bIsPlaying: false, bIsPaused: false, TotalFrames: 0, FramesRemaining: 0, TotalDuration: 0.000, TimePlayed: 0.000, TimeRemaining: 0.000, TotalPausedDuration: 0.000
ConvaiChatbotComponentLog: Character interrupted by user speaking
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“final”:true,“text”:“please follow me”,“timestamp”:“2026-05-15 19:50:16”,“user_id”:“PA_Misp2UReTMB9”},“label”:“rtvi-ai”,“type”:“user-transcription”}
ConvaiPlayerLog: Transcription received: please follow me
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“label”:“rtvi-ai”,“type”:“user-stopped-speaking”}
ConvaiChatbotComponentLog: Character interrupt ended
ConvaiPlayerLog: Finished talking
ConvaiPlayerLog: Transcription received:
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“event_type”:“user_text_message”,“extras”:{“text”:“please follow me”},“message”:“User text message processed”,“status”:“success”,“type”:“server-response”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Warning: OnDataPacketReceived: Unknown server type ‘server-response’.
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“participant_id”:“PA_Misp2UReTMB9”,“text”:“please follow me”,“type”:“final-user-transcription”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Warning: OnDataPacketReceived: Unknown server type ‘final-user-transcription’.
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“label”:“rtvi-ai”,“type”:“bot-llm-started”}
ConvaiFaceSyncLog: ConvaiStopLipSync: Stopping lipsync - bIsPlaying: false, bIsPaused: false, TotalFrames: 0, FramesRemaining: 0, TotalDuration: 0.000, TimePlayed: 0.000, TimeRemaining: 0.000, TotalPausedDuration: 0.000
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:“{”},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:" “},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:” “},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:” “”},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:“actions”},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:“”:“},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:” [ “},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:” “},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:” { “},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:” “},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:” “”},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:“name”},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:“”:“},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:” “”},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:“Follow”},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:“”,“},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:” “},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:” “},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:” “”},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:“target”},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:“”:“},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:” “”},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:“User”},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:“”“},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:” “},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:” “},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:” } “},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:” “},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:” ] “},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:”}“},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:” “},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“label”:“rtvi-ai”,“type”:“bot-llm-stopped”}
ConvaiChatbotComponentLog: UConvaiChatbotComponent Transcription received: | Character ID : 89649c94-4a21-11f1-935d-42010a7be02e | Session ID : -1
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“label”:“rtvi-ai”,“type”:“bot-tts-started”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“label”:“rtvi-ai”,“type”:“bot-tts-stopped”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:”{ “actions”: [ { “name”: “Follow”, “target”: “User” } ] } “},“label”:“rtvi-ai”,“type”:“bot-tts-text”}
ConvaiFaceSyncLog: ConvaiPauseLipSync: Pausing lipsync - TotalFrames: 0, FramesRemaining: 0, TotalDuration: 0.000, TimePlayed: 0.000, TimeRemaining: 0.000, TotalPausedDuration: 0.000
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“label”:“rtvi-ai”,“type”:“bot-started-speaking”}
ConvaiAudioStreamerLog: New SampleRate: 48000
ConvaiAudioStreamerLog: New Channels: 1
ConvaiAudioStreamerLog: [AudioStreamer] ProcessIncomingAudio - Starting Lipsync
ConvaiSubsystemLog: Active speaker changed: convai-bot
ConvaiSubsystemLog: [BlendshapeTurnStats] Server: 194 frames | Received: 194 frames | Match: YES | Audio: 155200 bytes | Duration: 2.95 s | FPS: 60.00
ConvaiSubsystemLog: Active speaker changed: convai-bot
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“label”:“rtvi-ai”,“type”:“bot-stopped-speaking”}
ConvaiChatbotComponentLog: UConvaiChatbotComponent Transcription received: | Character ID : 89649c94-4a21-11f1-935d-42010a7be02e | Session ID : -1
ConvaiSubsystemLog: Active speaker changed: convai-bot
ConvaiFaceSyncLog: Tick: Can’t play lipsync - CurrentSequenceTimePassed: 3.235405 - MainSequenceBuffer.Duration: 3.233334
ConvaiFaceSyncLog: Tick: Can’t play lipsync - CurrentSequenceTimePassed: 3.243760 - MainSequenceBuffer.Duration: 3.233334
ConvaiFaceSyncLog: Tick: Can’t play lipsync - CurrentSequenceTimePassed: 3.251841 - MainSequenceBuffer.Duration: 3.233334
ConvaiFaceSyncLog: Tick: Can’t play lipsync - CurrentSequenceTimePassed: 3.260251 - MainSequenceBuffer.Duration: 3.233334
ConvaiFaceSyncLog: Tick: Can’t play lipsync - CurrentSequenceTimePassed: 3.268399 - MainSequenceBuffer.Duration: 3.233334
ConvaiFaceSyncLog: Tick: Can’t play lipsync - CurrentSequenceTimePassed: 3.277012 - MainSequenceBuffer.Duration: 3.233334
ConvaiFaceSyncLog: Tick: Can’t play lipsync - CurrentSequenceTimePassed: 3.285125 - MainSequenceBuffer.Duration: 3.233334
ConvaiFaceSyncLog: Tick: Can’t play lipsync - CurrentSequenceTimePassed: 3.293714 - MainSequenceBuffer.Duration: 3.233334
ConvaiChatbotComponentLog: onAudioFinished - IsConnectionTalking: false
ConvaiFaceSyncLog: ConvaiPauseLipSync: Pausing lipsync - TotalFrames: 194, FramesRemaining: 0, TotalDuration: 3.233, TimePlayed: 3.294, TimeRemaining: 0.000, TotalPausedDuration: 0.000
ConvaiSubsystemLog: [BlendshapeTurnStats] After 2.0s inactivity - Server: 194 | Final Received: 194 | Late arrivals: 0 | Match: YES
LogViewport: Display: Viewport MouseLockMode Changed, LockOnCapture → LockAlways
LogViewport: Display: Viewport MouseCaptureMode Changed, CapturePermanently → CaptureDuringMouseDown
LogViewport: Display: Viewport MouseLockMode Changed, LockAlways → LockOnCapture
LogViewport: Display: Viewport MouseCaptureMode Changed, CaptureDuringMouseDown → CapturePermanently
ConvaiFaceSyncLog: ConvaiStopLipSync: Stopping lipsync - bIsPlaying: true, bIsPaused: true, TotalFrames: 194, FramesRemaining: 0, TotalDuration: 3.233, TimePlayed: 3.294, TimeRemaining: 0.000, TotalPausedDuration: 0.000
ConvaiChatbotComponentLog: Character interrupted by user speaking
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“label”:“rtvi-ai”,“type”:“bot-llm-started”}
ConvaiAudioStreamerLog: [AudioStreamer] MarkEndOfAudio - Total audio received from WebRTC: 314880 bytes, 3.280 seconds (SR: 48000, CH: 1)
ConvaiFaceSyncLog: ConvaiStopLipSync: Stopping lipsync - bIsPlaying: false, bIsPaused: false, TotalFrames: 0, FramesRemaining: 0, TotalDuration: 0.000, TimePlayed: 0.000, TimeRemaining: 0.000, TotalPausedDuration: 0.000
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“label”:“rtvi-ai”,“type”:“user-started-speaking”}
ConvaiPlayerLog: Started talking
ConvaiFaceSyncLog: ConvaiStopLipSync: Stopping lipsync - bIsPlaying: false, bIsPaused: false, TotalFrames: 0, FramesRemaining: 0, TotalDuration: 0.000, TimePlayed: 0.000, TimeRemaining: 0.000, TotalPausedDuration: 0.000
ConvaiChatbotComponentLog: Character interrupted by user speaking
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“final”:true,“text”:“please follow me”,“timestamp”:“2026-05-15 19:50:49”,“user_id”:“PA_Misp2UReTMB9”},“label”:“rtvi-ai”,“type”:“user-transcription”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“label”:“rtvi-ai”,“type”:“user-stopped-speaking”}
ConvaiChatbotComponentLog: Character interrupt ended
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“event_type”:“user_text_message”,“extras”:{“text”:“please follow me”},“message”:“User text message processed”,“status”:“success”,“type”:“server-response”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiPlayerLog: Transcription received: please follow me
ConvaiSubsystemLog: Warning: OnDataPacketReceived: Unknown server type ‘server-response’.
ConvaiPlayerLog: Finished talking
ConvaiPlayerLog: Transcription received:
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“participant_id”:“PA_Misp2UReTMB9”,“text”:“please follow me”,“type”:“final-user-transcription”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Warning: OnDataPacketReceived: Unknown server type ‘final-user-transcription’.
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“label”:“rtvi-ai”,“type”:“bot-llm-started”}
ConvaiFaceSyncLog: ConvaiStopLipSync: Stopping lipsync - bIsPlaying: false, bIsPaused: false, TotalFrames: 0, FramesRemaining: 0, TotalDuration: 0.000, TimePlayed: 0.000, TimeRemaining: 0.000, TotalPausedDuration: 0.000
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:“I’m”},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:” already"},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:" following"},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:" you"},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:“.”},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:“{ “actions”: [ { “name”: “Follow”, “target”: “User” } ] } I’m already following you.”},“label”:“rtvi-ai”,“type”:“bot-transcription”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“label”:“rtvi-ai”,“type”:“bot-tts-started”}
ConvaiChatbotComponentLog: UConvaiChatbotComponent Transcription received: { “actions”: [ { “name”: “Follow”, “target”: “User” } ] } I’m already following you. | Character ID : 89649c94-4a21-11f1-935d-42010a7be02e | Session ID : -1
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:" Just"},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:" lead"},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:" the"},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:" way"},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:“,”},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:" and"},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:" I’ll"},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:" keep"},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:" up"},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:“!”},“label”:“rtvi-ai”,“type”:“bot-llm-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:" Just lead the way, and I’ll keep up!“},“label”:“rtvi-ai”,“type”:“bot-transcription”}
ConvaiChatbotComponentLog: UConvaiChatbotComponent Transcription received: Just lead the way, and I’ll keep up! | Character ID : 89649c94-4a21-11f1-935d-42010a7be02e | Session ID : -1
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“emotion”:“neutral”,“scale”:2,“type”:“bot-emotion”},“label”:“rtvi-ai”,“type”:“server-message”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“label”:“rtvi-ai”,“type”:“bot-llm-stopped”}
ConvaiChatbotComponentLog: UConvaiChatbotComponent Transcription received: | Character ID : 89649c94-4a21-11f1-935d-42010a7be02e | Session ID : -1
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“label”:“rtvi-ai”,“type”:“bot-tts-stopped”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“label”:“rtvi-ai”,“type”:“bot-tts-started”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:“I’m already following you.”},“label”:“rtvi-ai”,“type”:“bot-tts-text”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“label”:“rtvi-ai”,“type”:“bot-tts-stopped”}
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“data”:{“text”:” Just lead the way, and I’ll keep up!"},“label”:“rtvi-ai”,“type”:“bot-tts-text”}
ConvaiFaceSyncLog: ConvaiPauseLipSync: Pausing lipsync - TotalFrames: 0, FramesRemaining: 0, TotalDuration: 0.000, TimePlayed: 0.000, TimeRemaining: 0.000, TotalPausedDuration: 0.000
ConvaiSubsystemLog: Attendee ID: convai-bot, Data: {“label”:“rtvi-ai”,“type”:“bot-started-speaking”}
ConvaiAudioStreamerLog: [AudioStreamer] ProcessIncomingAudio - Starting Lipsync

[Nicolas_Leurquin]

i changed the AI LLM model, now she does follow me. Changed from gpt-realtime-1.5 (beta) to gpt-4o-mini

[K3]

Great, now could you please test on XR level?

[Nicolas_Leurquin]

Yes, I will do that, I believe I already checked before, I changed the LLM earlier to get faster response time after already testing with the gpt 4o if I remember correctly. Will tell you in a minute

[Nicolas_Leurquin]

I also struggled with her sometimes not interacting/listening with that LLM model, as well as now. It seems the NPC is disconnected on the convai dashboard. She would sometimes suddenly listen and talk back after i had let the application run for multiple minutes

[K3]

I’m not fully sure I understand the issue.

Please note that if there is no interaction for around 5 minutes, the character become disconnected. If that is what you’re seeing on the Convai Dashboard.

[Nicolas_Leurquin]

No, in the application, it says connecting, then connected. However if I talk to the NPC after “connected”, she doesn’t interact or listen or anything, just idle animation. On the convai dashboard I noticed that she stays disconnected. It seems like they are not synchronized or something

[Nicolas_Leurquin]

i will perhaps try another LLM model

[Nicolas_Leurquin]

I reopened the project and tested with the realtime LLM, she does answer right away after it says connected. However when saying follow me she says the earlier structure: player action name…

[Nicolas_Leurquin]

Ok I managed to get interactions with gpt 4o mini, when saying follow me, she goes to the VRPawn origin again, and stays there, she doesnt follow me in VR

[Nicolas_Leurquin]

isn’t the difference the way location of the player works in the first person template compared to the VRTemplate?