LLMs, while powerful, are limited by their training data. Retrieval mechanisms are essential for grounding their responses in external knowledge. Two key approaches are Document Search and Graph Traversal. Document search retrieves relevant text based on keyword or semantic similarity, while graph traversal uses knowledge graphs to navigate relationships between concepts. This piece compares these methods, focusing on GraphRAG pipelines, and Notebook LM - a document search tool that works like a RAG system developed by Google, to guide the selection of appropriate knowledge augmentation strategies.

Following is a table for a summarised, direct comparison between GraphRAG and Notebook LM:

‍

The mechanics behind document search

Notebook LM by Google is an example of an enterprise level LLM-powered document search implementation. It's amazing and is reminiscent of how we develop popular RAG frameworks. The way document search works is by locating relevant information based on user queries - involving:

• Indexing: Preprocessing documents (tokenisation, etc.) and creating a searchable data structure. There are many techniques that can be used to do this such as TF-IDF (weighs words based on their frequency and rarity in a document and greater collection), BM25 (improves TF-IDF by considering document length), and Embeddings (represent words/documents as vectors, enabling semantic similarity comparisons).
• Querying: Matching processed queries against the index using keyword matching or comparing embeddings.

Document search solutions are easy to implement with readily available tools and is a mature architecture with well-established and optimised algorithms working in the backend. However, it has its limitations such as:

• Limited relationship capture: Struggles with complex connections between concepts. GraphRAG systems excel in these tasks, as they explicitly model relationships. It should also be noted that searches work primarily through keywords, and therefore can miss semantically related information in architectures due to absence of embeddings.
• LLM context window restrictions: Processing large documents and context is challenging and naturally requires higher computational costs.

Mechanics of graph traversal

Graph traversal offers an alternative approach by representing knowledge as a network of interconnected entities (nodes) and relationships (edges). This structure allows for more sophisticated retrieval based on the connections between concepts.

• Knowledge graph construction: Building a knowledge graph involves identifying key entities and the relationships between them. This can be done manually, automatically using information extraction techniques, or through a combination of both. The quality and completeness of the knowledge graph are crucial for the effectiveness of graph traversal. Being a dataset building outfit, the Knowledge Graphers and domain experts at Acme AI do this very well which makes our LLM applications work so well in the grassroots.
• Graph traversal algorithms: Once the graph is constructed, various algorithms can be used to traverse it and retrieve relevant information. Common algorithms include breadth-first search, depth-first search, and graph embeddings. These algorithms determine how the graph is explored to find the most relevant nodes and relationships.
• Querying the graph: Queries are translated into graph traversal operations, navigating from one node to another based on the specified relationships. For example, a query like "Who is the COO of Acme AI?" would translate into traversing the graph from the "Acme AI" node to the "COO" relationship and retrieving the connected node (the COO's name).

Graph traversal offers key advantages in regards to modelling complex relationships and enabling nuanced retrieval - which it extracts connections, not keywords, making it less vulnerable to parsing variations. Exploring relationships also uncovers insights missed in document search - using which LLMs can provide responses of higher quality. However, like document search, GraphRAG pipeline has some limitations:

• Graph structure: Building and maintaining a quality knowledge graph is challenging and impacts result quality - a problem that fuels the raison d'être for companies like Acme AI.
• Scalability: Traversing large graphs can get expensive based on the use-case and languages where tokenisation range varies substantially. Our experiments in GPT 4o showed tokens for Bengali costing 8x more than the English counterpart which would be further compounded by the complexity of the graph.

Real-world example: LLM-powered Public Policy Analysis

Let’s work with an example surrounding computational public policy analysis. Let's consider the analysis of public policy related to climate change.

Notebook LM

If you uploaded documents about the Paris Agreement, the automotive industry, and the EU's regulations, Notebook LM could potentially identify connections if these topics are discussed within the same documents or if there are clear textual overlaps. However, if the documents are separate and don't explicitly link these concepts, Notebook LM might not make the connection as effectively. It would rely on the user to ask more specific questions that draw these connections out. It would not be able to readily answer a question like "What is the indirect impact of the Paris Agreement on the automotive industry in the EU via policy X and regulation Y" if those intermediary steps are not explicitly written out in the uploaded documents.

GraphRAG Pipeline

A GraphRAG approach would represent the knowledge as a graph. Nodes could represent:  

• Policies (e.g., "Paris Agreement," "Clean Power Plan")
• Organisations (e.g., "United Nations," "Environmental Protection Agency")
• Concepts (e.g., "carbon emissions," "renewable energy," "economic impact")
• Geographic locations (e.g., "European Union," "United States")

Edges could represent relationships like:

• "Implements" (e.g., "United Nations" implements "Paris Agreement")
• "Regulates" (e.g., "Environmental Protection Agency" regulates "carbon emissions")  
• "Impacts" (e.g., "Paris Agreement" impacts "economic growth")
• "Located in" (e.g., "European Union" located in "Europe")

A query like "What is the impact of the Paris Agreement on the automotive industry in the European Union?" could be answered by traversing the graph: starting at the "Paris Agreement" node, following the "impacts" edge to related concepts, then filtering by the "automotive industry" and "European Union" nodes. This provides a much more direct and insightful answer than document search, which would require sifting through numerous documents to find the relevant connections.

A question you might have is ‘can Notebook LM not relate to different docs?’. You are right to ask this question as did we when exploring both architectures. Notebook LM connects information across documents within a notebook through textual analysis, using semantic search and key phrase identification. It infers relationships based on word proximity and co-occurrence, summarising relevant sections from multiple documents. However, unlike GraphRAG, Notebook LM's understanding of relationships is implicit, relying on textual patterns rather than explicitly defined connections in a knowledge graph. This limits its ability to perform complex reasoning, understand the nature of relationships, and connect distantly related concepts. GraphRAG's explicit relational modeling offers more robust and precise connections.

What to use and where?

The choice between GraphRAG and document search depends heavily on the specific use case and the nature of the data.

GraphRAG is well-suited for knowledge-intensive tasks - where understanding of relationships and connections is crucial. Use-cases include drug discovery and its connectivity to genes and diseases, financial analysis such as market trends and economic indicators, supply chain management and its flow of goods and identification of potential disruptions based on historical insights, and complex Q&A which requires reasoning over multiple facts and relationships - an example being our Probahini Menstrual Health Chatbot co-designed with WaterAid and the MHM Platform. It is also appropriate for use-cases where information changes are frequent and need to be updated in a structured way - as knowledge graph updates are fundamentally more efficient than re-indexing large document collections.

Document search systems such as Notebook LM are better suited for document analysis and summarisations. In these cases, extracting specific information from documents hold precedence such as legal document review to analyse contracts and legal precedents, research paper summarisation to understand key findings and methodologies, personal knowledge management that works to organise and retrieve information from personal notes. It also enables exploratory search where a user is unsure of the exact information they are looking for and want to browse through relevant documents.

In essence, if your data is highly relational and your queries require reasoning over those relationships, GraphRAG is likely the better choice. If your focus is on analyzing and summarizing existing documents, document search is generally more appropriate. Upto this point, we explored two distinct approaches for grounding LLMs in external knowledge: Graph Traversal, as exemplified by the GraphRAG architecture, and Document Search, commonly used in systems like Notebook LM. We've seen how Graph RAG excels at capturing complex relationships, offering robust retrieval, and uncovering hidden connections, while Document Search provides a simpler and more efficient way to analyze and summarize existing documents. There are also hybrid experimentations that are possible and on-going in different pockets.

Acme AI works in both spectrums - offering support to build knowledge graphs and deconstructing large volume of documents for document search ingestion for a wide variety of use-cases.

We work with clients to activate supervised finetuning (SFT), reinforcement learning from human feedback (RLHF) and AI feedback (RLAIF), co-develop autonomous chatbots, and co-design LLM pipelines. Connect with us if you have a use-case at info@acmeai.tech.