Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 6 min read

Why Does RAG Need a Vector Database If LLMs Can Already Understand Text?

If you are learning about Generative AI and RAG, you might have come across embeddings and vector databases. But here is a question that might confuse beginners: If we eventually send text to an LLM to get an answer, w

Why Does RAG Need a Vector Database If LLMs Can Already Understand Text?

If you are learning about Generative AI and RAG, you might have come across embeddings and vector databases.

But here is a question that might confuse beginners:

If we eventually send text to an LLM to get an answer, why do we need to convert that text into embeddings and store them in a database? Why can't we just send the text directly to the LLM?

Before answering that, let's understand how RAG works.

1. What is RAG?

RAG stands for Retrieval-Augmented Generation.

In simple words, RAG allows an AI model to answer questions using information from our own documents, rather than relying only on what it already knows.

For example, imagine you have built an AI assistant for a company. You have uploaded hundreds of documents containing product information, user manuals, FAQs and company policies.

Now, an employee asks:

"How can I reset my password?"

Instead of expecting the AI model to already know your company's password reset process, RAG finds the relevant information from your documents and gives it to the AI model to generate an answer.

2. What happens when someone asks a question?

Let's understand the basic flow.

  1. A user asks a question: "How can I reset my password?"
  2. The system searches your stored documents to find information related to that question.
  3. It retrieves the most relevant text.
  4. That text, along with the original question, is sent to the LLM.
  5. The LLM reads the information and generates an answer.

For example, the LLM might respond:

"To reset your password, visit the login page, click Forgot Password, and follow the instructions sent to your registered email."

That's RAG in action!

3. But why not send all documents directly to the LLM?

Now comes the interesting part.

Imagine your company has 10,000 documents.

When someone asks about resetting a password, why should we send all 10,000 documents to the LLM?

Most of those documents are probably irrelevant to the question.

Sending everything creates a few problems:

  • Cost: More text means more tokens, which increases API costs.
  • Time: Processing thousands of documents for every question takes longer.
  • Limits: LLMs have a limit on how much text they can process in a single request.
  • Unnecessary information: The LLM receives lots of information it doesn't need.

So, instead of sending everything, we need a way to find only the relevant information.

4. What are embeddings, and how do they help?

Let's take a simple example.

A user asks:

"How can I reset my password?"

But somewhere in your documentation, the actual text says:

"Steps to recover access when you forget your login credentials."

The words are different, but both sentences have almost the same meaning.

A normal keyword search might not find a good match.

This is where embeddings help.

An embedding is a numerical representation of text that helps a computer understand and compare its meaning.

For example, sentences with similar meanings generally have similar embeddings, even if they use different words.

This allows us to search for information based on meaning rather than just matching keywords. This approach is called Semantic Search.

5. Why store embeddings in a database?

Before users start asking questions, we prepare our documents.

The process looks like this:

  1. Take all the documents.
  2. Break large documents into smaller pieces (called chunks).
  3. Convert each chunk into an embedding using an embedding model.
  4. Store the original text, its embedding and useful information (metadata) in a database.

Now, when someone asks a question:

  1. Convert the question into an embedding.
  2. Search the database using the question's embedding to find similar content (this is where semantic search happens).
  3. Retrieve the most relevant original text.
  4. Send that text along with the user's question to the LLM.
  5. Generate the answer.

One important thing to understand: We don't send embeddings to the LLM to generate the answer. We send the original text. Embeddings are used to find the right text.

Why not generate embeddings every time?

Imagine you have 10,000 documents.

If you generate embeddings for all 10,000 documents every time someone asks a question, you are repeating the same work again and again.

Instead, we generate document embeddings once and store them.

When a new question arrives, we only generate an embedding for that question and search the existing embeddings.

If a document changes, we update its corresponding chunks and embeddings.

This makes the process faster and more efficient.

6. How do we get the original text back?

When we search the database using embeddings, we get the matching records, not just the embedding numbers.

Why? Because we store the original text and its embedding together in the database.

For example, a record might contain:

  • Text: Employees are entitled to 20 days of annual leave.
  • Embedding: [0.12, 0.45, 0.78, ...]

When a search finds a matching embedding, the database returns its associated original text.

We then send that text to the LLM along with the user's question.

In simple words, embeddings help us find the text, and the database gives us the original text back.

7. Why not send embeddings directly to the LLM?

If embeddings represent the meaning of text, why can't we simply send those numbers to the LLM and ask it to generate an answer?

The reason is simple: Embeddings are designed for searching and comparing meaning, not for reading and understanding the original information.

Let's take an example.

Suppose your document contains:

"Employees are entitled to 20 days of annual leave."

An embedding model might convert this sentence into hundreds or thousands of numbers, such as:

[0.023, -0.145, 0.876, 0.321, ...]

These numbers help us find similar content, but they don't contain the original sentence in a form the LLM can directly read and use.

The LLM needs the actual information:

"Employees are entitled to 20 days of annual leave."

So, embeddings help us find the information, but the original text helps the LLM answer questions.

8. Why send the question along with the retrieved text?

Another interesting question!

If we have already searched the database and found the relevant document, why do we need to send the original question again? Can't we just send the document to the LLM?

Let's understand with an example.

Imagine the retrieved document contains:

"Employees are entitled to 20 days of annual leave. Sick leave is limited to 10 days per year."

Now, consider two different questions:

  • "How many annual leave days do employees get?"
  • "How many sick leave days do employees get?"

The retrieved document is the same, but the answers are different.

If we send only the document, the LLM doesn't know which information the user is actually looking for.

By sending both the question and the retrieved text, we give the LLM the necessary context:

Question: How many annual leave days do employees get?

Retrieved information: Employees are entitled to 20 days of annual leave. Sick leave is limited to 10 days per year.

Expected answer: Employees get 20 days of annual leave.

The question tells the LLM what the user wants to know, while the retrieved text provides the information needed to answer it.

That's why both are important.

9. What does each component actually do?

Let's keep it simple:

  • Embedding model: Converts text into numbers so we can compare meanings.
  • Vector database: Stores these numbers and helps us find similar content.
  • LLM: Reads the retrieved text and generates a human-readable answer.

Final thoughts

The purpose of RAG is not to help an LLM understand text. LLMs can already do that.

The real purpose is to find the right information from a large collection of documents before sending it to the LLM.

Think of it this way:

  • Embeddings help us find the right information.
  • The vector database stores and searches that information.
  • The original question tells the LLM what we want to know.
  • The retrieved text provides the information.
  • The LLM uses both to generate an answer.

We don't store embeddings because the LLM needs them to answer. We store them because we need an efficient way to find the right information before asking the LLM.

This post was initially published on my blog. Check out the original source using the link below:

Why Does RAG Need a Vector Database If LLMs Can Already Understand Text?

Ever wondered why we store embeddings in a vector database in RAG when we eventually send text to an LLM? In this article, I explain how RAG works, why…

favicon rahulvijayvergiya.hashnode.dev
πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.