Issue with Knowledge Base Behavior

Core Features Knowledge Bank
1

I am experiencing an issue with the knowledge base of your service.

I have several documents that I need to use for an avatar we are developing for a museum. When I upload these documents to a service like GPT-4o or Google’s Notebook LM, these systems correctly respond even to highly specific questions.

However, when querying the same documents (using GPT-4o as the model) within ConvAI, the behavior is highly inconsistent. Often, the responses fail to capture even the most prominent details.

The character ID I am using is 09f85624-eea8-11ef-85b7-42010a7be016, and I have currently uploaded several documents to the knowledge base, of which only two have been activated: one of 1.57KB and one of 648.69KB. The imprecise behavior remains the same even when working with just a single document.

Could you help me understand if I am doing something wrong?
Is it correct to expect that responses using GPT-4o with ConvAI should be the same as those obtained by directly using GPT-4o and the “project” feature as RAG?

Note:
Someone in this forum says that knowledge base supports English language only, but I can’t find any reference anywhere stating that the knowledge base only works in English. Could you please confirm this point?

It seems really strange to me that the knowledge base would only work in English… why?
I have already used the knowledge base in Italian for another project in December 2024, and it worked. Moreover, a RAG that uses embeddings to index documents should be language-independent.

Additionally, the behavior I am observing with this avatar is inconsistent: sometimes it responds correctly… other times it does not.

There is one aspect of your knowledge base document guidelines that I don’t fully understand. You mention using a “single file format.”
We have a PRO account with a 20MB limit for the knowledge base. Is it better to upload a single 20MB file rather than 20 files of 1MB each?

2

I have similar issue too.

I set the AI to gpt-4o-mini with a temperature of 0.25, so it should achieve responses that are more precise, consistent, and less random.

:eyes:

3

Yeh as Francis_Perron has mentioned, best to set your temperature to very low or maybe 0.

According to the docs it’s recommended to use a single text file, I assume even if that’s not possible then to use a minimal amount would be better. I think it is recommended to group the info you have in the text file(s) into paragraphs, each paragraph being the subject of 1 main thing, and iirc have say no more than 4 or 5 sentences in each paragraph. The docs can confirm most of this I think. Here is a sample paragraph from 1 of my Knowledge Bank files :

In the latest news, Russia’s war in Ukraine is still ongoing. President Trump has entered into negotiations with to President Putin about reaching a peace deal. There was a major rift between Ukraine President Zelensky and President Trump. The 2 men were trying to agree a deal that would see some of Ukraine’s valuable raw mineral resources being exchanged for America’s support of Ukraine’s war effort via money and miliary equipment. Ukraine has agreed to Trumps 30 day cease fire proposal although a cease fire is not yet in place because Russia’s Putin is cagey about a ceasefire.

Edit to add : my Knowledge Bank works really well tbh.

4

Are you working in english or in other languages?

Anyone that experienced success use case in foreign languages (such as italian, french, spanish or other different from english)?

5

My tests were with french texts within multiple files.
I will try to merge them into one file to see if its better and lower the temperature if needed.

1 Like
6

@omar.venturi : Can you please confirm. if you are still facing the issue?

7

I can confirm that it works in italian.
Anyway it’s still not so precise as “standard” llms are.
It can’t find specific information as GPT-4o (for instance) does.

8

I see. We have a new feature. Would you be interested in giving it a try. The feature should better understand your document structure for AI response. If you are willing I can enable it for you to try

9

it could be interesting to try it.
The avatar I working on is almost “in production” (starting on May 3rd) and has been tested and optimized based on current ConvAI behaviour.
Is it possible for instance to try this new feature in a separate avatar?

Anyway we could test it on a differente account (we have more than one account)

10

@omar.venturi , Could you please share the email address of your other account?

11

Sure, but I’d prefer not to share it here. Could you give me an email address I can send it to?

12

I sent you a PM message.

18

I am also struggling with the KB and it appears from the messages here and elsewhere on the forum that this is a problem across the board. I’m also frustrated that there appears to be some sort of feature that you’ve mentioned here and in other conversations that ‘better understand your document structure for AI response’ which appears to only be available to people who write posts saying that there is a problem. Why is this feature not implemented if it helps the system perform better - especially to those that are on a paid subscription?

I have found, for my purposes, that combining information into a single txt document has elicited better results than having multiple separate documents but I’ve still come unstuck on a number of occasions because the avatar seems to not want to directly quote information from the KB but rather summarise. I have been previously told that the files within the KB cannot be directly quoted by my avatar in their responses due to a security measure you have implemented - is this still the case? Can this be turned off? I would be happy as a user of your Product to have this decision rest on my shoulders as the user and make that security decision myself.

Other freely available LLMs (and indeed many of the ones that are available within the Convai ‘Core AI Settings’) will directly quote multiple documents accurately but within the confines of the CONVAI ecosystem this seems to break down.

In order to bring products to market we need this functionality to be consistent, reliable and accurate.

19

Hello @Academii_Ltd,

Welcome to the Convai Developer Forum!

Thank you for your thoughtful message and for taking the time to share your detailed feedback, it’s truly appreciated.

You’re absolutely right to expect consistent, reliable, and accurate Knowledge Bank functionality, especially as you work toward bringing a product to market. The feature you’ve mentioned, is currently in beta and still undergoing internal testing. That’s why it isn’t yet widely available. We want to ensure it performs reliably across a range of use cases before rolling it out to all users.

That said, we’d be happy to enable this beta feature for your account to help with your current challenges. If you’re using the same email associated with this post, we can proceed with activating it for you. Just let us know.

In the meantime, please make sure:

  • Your Knowledge Bank documents are in English (as this currently yields the most consistent results).
  • Use a lower temperature value (we suggest setting it to 0)
20

From what I’ve experienced, the ability of ConvAI’s knowledge bank to find the proverbial “needle in a haystack” is inferior to what can be achieved when working directly with LLMs.
This is the case even when documents are highly optimized, temperature is set low, and different LLMs are tested (assuming the LLM plays a role at all… which I personally doubt, since in a typical RAG setup, it’s the retriever — not the LLM — that’s responsible for fetching results).

That said, I’ve observed significant improvements by refining the prompt and, most importantly, by improving the quality of the uploaded data.

My suggestion is to create a new, “clean” avatar (with a basic prompt such as “you are a virtual assistant” and nothing more), and then start preparing the content according to ConvAI’s guidelines.

This “clean avatar” approach helped me understand that sometimes (at least in my case), responses weren’t actually coming from the knowledge bank but rather from the LLM’s native training — likely because the KB contained information about a very well-known historical figure.
By using a minimal prompt and explicitly instructing the avatar to ignore everything except what’s in the KB, I was indeed able to get more specific responses.
I even managed (as a test) to make it provide false answers — for example: “Who discovered America?” → “Charlemagne.”

If a strict “question-answer” format is not feasible (which is often the case), I recommend at least pre-processing the material through manual chunking.
That means deciding where and how much to split sentences and how much overlap to include between chunks — these are technical aspects that come into play when preparing content for systems like ConvAI’s knowledge bank. (Search online for “chunking RAG best practices” to dig deeper.)

Also, use as few documents as possible (this is even mentioned in ConvAI’s official guidelines). In general, it’s better to have a few large documents than many small ones.
[Honestly, I haven’t tested this part much myself, and it seems counterintuitive that it would have much impact — but I’m including it for completeness.]

This has been my experience. The quality has definitely improved compared to the initial results…
but I still haven’t been able to achieve the same level of quality — for example — as with an agent built using ChatGPT.

21

Thanks for this inciteful response. I’ll have a go at your suggestions along with the help from CONVAI directly.

22

Hi K3 and thanks, as ever, for the swift response.

Yes please, I’d be grateful to try this beta feature and see if this improves things. I’d also be happy to report back on my findings to hopefully help with the development process.

I’ve lowered the temp but have never tried taking it down to 0 so will do that also, thanks.

Cheers,

23

Also, worth asking but is there some way of turning off whatever measure has been put in place to specifically stop the Avatars from sharing verbatim information from documents in the KB?

25

Hello @omar.venturi, @Academii_Ltd,

The changes have been activated for your accounts. Please reupload the files and attach to character again.