204: Implement RAG Use Cases

Vector databases

The "knowledge base" for the retrieval step is typically implemented as a vector database. Vector databases are not new, they have been used for several years for semantic search.

Semantic search is a search technique that aims to understand the intent and context of a user's query. It's more advanced than traditional keyword-based search, which relies on matching specific keywords.

Some popular vector databases are:

• Chroma (open-source)
• Milvus
• Elasticsearch
• SingleStore
• Pinecone
• Redis
• Postgres

Some vector databases, like _Chroma, are in-memory databases, which makes it a good choice for experimenting and prototyping RAG use cases. In-memory databases do not need to be installed or configured. This is because all of their data is stored in volatile memory for the duration of the program's runtime. The downside of this though is that we will need to load the data each time we run the application.

Vector databases store unstructured data in a numeric format. For AI use cases, we use a specific term to describe the converted unstructured data - embeddings. Embeddings are created by using an embedding model, for example, a popular open-source model _word2vec__.

One of the key features of embeddings is the ability to preserve relationships. With embeddings, words can be "added and subtracted" like vectors in math. One of the most famous examples that demonstrate this is the following:

king - man + woman = queen
txt
Copy code to clipboard

In other words, adding the vectors associated with the words king and woman while subtracting man is equal to the vector associated with the word queen. This example describes a gender relationship.

Another example might be:

Paris - France + Poland = Warsaw
txt
Copy code to clipboard

The vector difference between Paris and France here captures the concept of a capital city.

Here's an example of a semantic search for the word "education". Notice the "nearest point" words in the table. The measurement is in numbers because the search was performed using vector (numeric) data.

You can experiment further with this here.

Based on what we have discussed so far, we can make the following conclusions:

• We have a choice of vector databases for implementations of RAG use cases
• We have a choice of several embedding models for converting unstructured data to vectors
• Retrieval of relevant information is the first step in the RAG pattern, and the quality may be affected by the choice of a vector database and an embedding model.

Heads up! Parts of this section will be on the quiz.

In addition to the mentioned factors, loading data into the vector database requires "chunking", which is the process of dividing text into sections that are loaded into the database. There is no single guideline for defining chunks because the best size depends on the content we're working with.

For generative AI use cases, we typically start with _fixed-size chunks, which should be "semantically meaninguful". For example, if in your document each paragraph has approximately 200 words, convert the number of words to several tokens (may vary per LLM), and use this number as the starting chunk size. In addition to the size of the chunk, we also need to specify overlapping (the same information in more than one chunk).

Model providers should publish information on recommended chunk size and overlap percentage for specific use cases. However, even if this information is published, developers will still need to experiment to find the optimal chunk size. If you can't find information on the recommended overlap, you can start with 10 percent (specified in the format of "number of tokens", for example, 100 if your chunk size 1000).

To summarize, we should follow these steps when implementing the RAG use case:

1. Select a vector database. When working with watsonx.ai, we should confirm that the selected vector database is supported. Check IBM documentation for the latest updates.
2. Select the embedding model. Confirm that the embedding model works with the selected vector database.
3. Experiment with chunk size.

As we discussed earlier, the second part of the RAG use case, sending instructions to LLM, is similar to non-RAG patterns. Developers should pay special attention to token limits because retrieved content will be added to the prompt. If the token limit is exceeded on input, then the model will not be able to generate output. This check will need to be implemented as "custom code" in your LLM applications.

WML API does return errors if token size is exceeded.