Can Unity beta package with vision see rendered virtual scene?

Core Features Language and Speech
, ,
1

Hi there,

I’d been experimenting with the Unity beta package with vision, trying to get it to see the rendered VR scene in my Quest 3 instead of live camera feed in MR mode.

So I’d deactivated the [BuildingBlock] Passthrough gameobject in Unity(which disables passthrough mode in Quest) and set Track Source in the Convai Vision Publisher Meta component of Vision Publisher gameobject to “Source Screenshare” or “Source Screenshare Audio” while Capture Camera is already set to “CenterEyeAnchor”, hoping that this will enable the vision functionality to process the virtual scene instead.

However this doesn’t work - the AI doesn’t respond to voice prompts any more(seemingly deactivated).

So am I missing something here?

Thanks!

Melvin Eng.

2

Hi again,

Umm, any feedback here would be appreciated :slightly_smiling_face:

Cheers,

Melvin.

3

Please use the latest version.

4

Hi Kaan,

Thanks for getting back.

I’d perused the latest beta plug-in, but didn’t find any reference to the vision capability. (on the other hand, the Unreal beta plug-in still has it)

So why was it removed? I presume it will be reinstated soon?

Regards,

Melvin.

5

Hi again Kaan,

Umm, a little bump for the post above…:slightly_smiling_face:

Cheers,

Melvin.

8

Why can’t you use latest version? Could you please show the problem?

9

Hi Kaan,

The problem is: the UPM package is taking an indefinitely long time to install(pls see screenshot below). Not sure if it’s some access issue?

Thanks,

Melvin.

10

Screenshot 2026-03-25 111245
Screenshot 2026-03-25 111245885×447 11.1 KB

11

Could you please share the logs?

12

Hi Kaan,

Thanks for getting back.

I’ve finally managed to install the beta UPM package from the git URL(it was conflicting with my existing Convai package). So it’s Convai SDK for Unity v4.0.0. I’m actually quite confused because this seems to be similar to the Convai SDK for Unity v4.0.0 downloadable from Unity Asset Store?

Anyway I’ve imported and opened the lip sync sample scene. I don’t see anything that connects with your instructions above. Leaving aside the VR stuff, how do I configure the character to see anything(be it the virtual scene or camera feed), assuming it’s even supported? This seems to be markedly different from the Convai Vision Unity Beta plug-in as before?

Very confused,

Melvin.

13

Hi,

Uh, little bump here again…

Regards,

Melvin.

14

Hi, thanks for reaching out!

If you want to add vision in a scene, you can follow the steps below.

  1. There is a GameObject whose name is [Convai Manager] in that Add Component ConvaiRoomManager and select Connection Type to Video.

    image
    image565×682 53.6 KB

  2. That will add a child to that game object called ConvaiVisionRoot. There is a component called CameraVisionFrameSource set the Target Camera.

image
image557×737 52.7 KB

Now the character will what ever the things that your unity camera can see.

Please let us know if you face any issues

Regards,
Convai Unity Team.

15

Hi there,

Thanks for getting back! I’d been waiting quite a while for a response :sweat_smile:

I have an additional question: suppose I want the AI to see my webcam feed instead, how do I modify the existing setup?

Many thanks!

Melvin.

16

Hi Melvin_Eng

In the existing setup, you just need to remove “CameraVisionFrameSource” and use “WebcamVisionFrameSource.”
By doing this, you can use a webcam feed for vision in place of the Unity camera feed.

Thanks,
Convai Unity Team.

17

Hi again,

Thanks for the tip! :smiley:

Cheers,

Melvin Eng.