External API tools never fire on Pixel Streaming Meta Human (only built-in `look` injected) — is this expected?

Core Features Core API
1

Hi team,

I have a Meta Human character configured with several External API tools
(Python handle_event) and I cannot get any of them to fire when the character
is interacted with via Pixel Streaming (the standard playground avatar
view). Looking for confirmation whether this is a platform limitation or a
config I am missing.

Setup

  • Character: dental clinic receptionist (Meta Human, Pixel Streaming).
  • LLM: GPT-4o (verified in Mindview, not gpt-4.1-mini).
  • External API → 7 methods defined (identify_patient, check_availability,
    book_appointment, lookup_appointment, list_my_appointments,
    cancel_appointment, reschedule_appointment).
  • Each method’s Input Description JSON Schema is strictly valid:
    every key in properties is also listed in required (no optional keys
    left in properties — I confirmed this is mandatory after a previous issue).
  • Connect toggle is ON for every method.
  • Python implementation is the standard handle_event(method, params, ...)
    pattern that POSTs to my backend with a shared secret. The backend logs
    every incoming request.
  • CONVAI_EXTERNAL_SECRET placeholder is replaced with the real value in the
    Python code.
  • Backstory contains explicit instructions to call these tools by name, with
    examples and parameter formats.

Expected

When I say things like “Please book me an appointment for tomorrow at 13:00”
in the Pixel Streaming chat, the LLM should call
check_clinic_availability and then book_clinic_appointment via the
External API, and my backend should log those POST requests.

Actual

  • The AI claims it did the work (e.g. “I checked availability,
    10:30 is free”, “I booked your appointment FSA-1234”) — full hallucination.
  • My backend logs show zero requests from Convai’s IPs. Only the warm-up
    pings I run myself.
  • In Mindview the system prompt rendered for the LLM contains only the
    built-in look tool. None of my External API tool definitions are
    injected into the prompt.

Question

  1. Is it correct that Pixel Streaming / Meta Human playground does not
    inject External API tool definitions into the LLM context, by design?
    The Mindview output strongly suggests this — only look shows up.
  2. If so, is the supported path to use External API tools either
    Web Chat / Text Embed mode or the Unreal Engine SDK (custom
    integration), and Pixel Streaming is intentionally limited to built-in
    tools (look, etc.)?
  3. Is there any roadmap item to bring External API tools to Pixel Streaming
    Meta Humans? This would be a big unlock for voice-first business use cases
    (clinics, hotels, support).

If this is the expected behavior, I’d suggest adding a clear note in the
External API docs (“Not available in Pixel Streaming mode — use Web Embed or
Unreal SDK”). It would have saved me a couple of debugging days.

Thanks in advance!

2

Could you please test this in the Legacy Playground using the chatbox, not the video call?

3

Hi Kaan,

We’re deploying Convai MetaHuman characters on physical holobox /

holographic display hardware in clinic and hotel reception areas.

Customers walk up, talk to a photoreal AI receptionist, and the

receptionist needs to actually book appointments / look up rooms in

our backend.

**What works today:**

• External API tool calling in Legacy Playground chatbox :white_check_mark:

• External API tool calling via @convai/web-sdk WebRTC :white_check_mark:

• Pixel Streaming Embed photoreal MetaHuman (cinematic) :white_check_mark:

**What does NOT work (and is the blocker for us):**

• External API tool calling INSIDE a Pixel Streaming session :cross_mark:

In Pixel Streaming sessions, Mindview only ever shows the built-in

`look` tool. Our External API methods are configured and active

(validated by exercising them via chatbox), but they are never invoked

during a Pixel Streaming conversation — no POST reaches our backend

even when the user explicitly asks for actions.

Convai support confirmed (ticket reply): "Could you please test this

in the Legacy Playground using the chatbox, not the video call?"

which we read as: External API is not yet wired into the Pixel

Streaming pipeline.

**The use case**

For physical holographic / holobox hardware (and in fact any kiosk

deployment with strict visual fidelity requirements), Web SDK

browser-rendered avatars are not sufficient. Pixel Streaming is the

only path that hits cinematic quality on these displays. Without

External API in Pixel Streaming, the avatar can talk beautifully but

cannot actually do anything for the customer — which defeats the

business case.

**Questions for the Convai team**

1. Confirm officially: is External API in Pixel Streaming UNSUPPORTED

 today, or is there a configuration step we're missing?

2. Is this on the roadmap? Any beta / closed access?

3. Workaround? (Action API injection, sidecar Web SDK chat session

 sharing session_id, server-side conversation hook, etc.)

4. Avatar Studio Experiences specifically — do they respect the

 External API methods configured on the underlying character?

Anyone else hitting the same wall with kiosk / holobox / signage

deployments? Would love to hear how you’re solving it.

Thanks,

Özgür — AssistFlow

4

This is not related to Pixel Streaming.

It is currently not supported on the new WebRTC infrastructure, but we plan to add this support in the future.