Why Can't I Just Use ChatGPT to Understand My Project? An Introduction to RAG

I recently joined a software project that had been running for almost 12 years.

As part of the onboarding process, the team shared a lot of documentation with me: architecture documents, technical specifications, workflows, and other project-related information.

I started going through them. Honestly, it was overwhelming. There was so much information to read, and manually going through document after document wasn't exactly an exciting experience.

One day, I was having a conversation with one of the senior developers on the team.

Senior Developer: "Have you gone through the documentation and got a good understanding of the project?"

Senior Developer: "How are you reading all these documents?"

He then mentioned that the team was building an AI assistant to make the onboarding process smoother for new joiners.

Me: "So, can't I just use ChatGPT or Gemini for this?"

Senior Developer: "Not exactly. ChatGPT is a general-purpose LLM. It doesn't automatically know the internal details of our project. This is where RAG comes into play."

That conversation introduced me to a concept I wanted to understand better: Retrieval-Augmented Generation (RAG).

But why isn't a general-purpose LLM enough?

"How does authentication work in a Java application?"

It can explain authentication concepts, common approaches, and implementation patterns.

"Where is authentication implemented in our project?"

The answer depends on our actual source code and internal documentation.

An LLM may understand authentication very well, but that doesn't mean it knows how authentication is implemented in our project. It doesn't automatically have access to our private project information.

Of course, we can upload documents or paste information into ChatGPT. But imagine doing that for a project with thousands of files and hundreds of documents.

We need a better approach: retrieve only the information relevant to the question and provide it to the LLM.

RAG stands for Retrieval-Augmented Generation.

Retrieval: Finding relevant information from an external knowledge source.

Augmented: Providing that information along with the user's question.

Generation: Using an LLM to generate an answer based on the provided information.

In simple terms, RAG connects an LLM with external knowledge so it can answer questions using information that isn't necessarily part of its training data.

A basic RAG application has two major workflows:

Retrieving information and generating an answer.

Before our AI assistant can answer questions about the project, it needs access to the project's information.

This could include architecture documents, technical specifications, API documentation, and other internal resources.

We need to prepare this information so that it can be searched efficiently. This process is called data ingestion.

There are four important concepts involved:

First, we collect and load information from different sources into the system.

Imagine an architecture document containing 100 pages. We don't need to retrieve the entire document whenever someone asks a question.

Instead, we divide it into smaller, manageable pieces called chunks. This makes it easier to find specific information.

How can a system find information based on meaning rather than just matching exact words?

An embedding is a numerical representation of text that captures aspects of its meaning. It helps a system identify semantically similar pieces of information, even when they use different words.

Once we convert the chunks into embeddings, we need to store them somewhere searchable.

A vector database stores these embeddings along with their associated information, such as the original text and metadata.

Now our project knowledge is prepared and ready to be searched.

"Where is authentication handled in our project?"

Step 1: Convert the question into an embedding

The user's question is converted into a numerical representation, just like the document chunks.

The system searches the vector database to find chunks that are semantically similar to the question.

For example, it might retrieve relevant information from authentication documentation, login API documentation, or security configuration documents.

The RAG application combines two things:

The relevant information retrieved from the knowledge base.

The LLM uses the question and retrieved information as context to generate a response.

Instead of relying only on its general training knowledge, it now has access to information about our specific project.

Knowledge preparation (before users ask questions):

Project documents → Document loading → Chunking → Embeddings → Vector database

Question answering (when a user asks a question):

User question → Question embedding → Retrieve relevant chunks → Combine question and retrieved context → LLM → Generated answer

The important distinction is that knowledge preparation happens beforehand. We don't need to process all the project documents from scratch every time someone asks a question.

When new documentation is added or existing information changes, the knowledge base can be updated through the ingestion process.

We now understand the basic idea behind RAG: retrieving relevant external information and providing it to an LLM to generate an answer.

But does this mean every answer generated by a RAG application will be correct?

What happens when the system retrieves irrelevant information? Why do LLMs sometimes generate answers that aren't supported by the provided context? How can we make the system more reliable?

In the next article, we'll explore these challenges through concepts such as hallucinations, prompt engineering, context management, hybrid search, and reranking.

That's where we move beyond understanding RAG to understanding how to build more reliable RAG applications.