This article is part of our coverage of the latest in AI research.
Knowledge storage and retrieval are becoming important parts of large language model (LLM) applications. And while there have been tremendous advances in the field with retrieval-augmented generation (RAG), some limitations remain unaddressed.
Researchers from Ohio State University and Stanford University have introduced HippoRAG, a novel retrieval framework inspired by the hippocampal indexing theory of human long-term memory.
HippoRAG enables LLM applications to integrate dynamic knowledge more efficiently and to retrieve important information faster and more accurately. HippoRAG shows impressive performance on tasks such as multi-hop question-answering and can be very useful for LLM applications that require assembling and reasoning over large and complex knowledge graphs.
The challenges of knowledge integration in LLMs
The mammalian brain can store vast amounts of knowledge and continuously integrate new experiences without losing previous ones. This long-term memory system allows humans to continuously update the knowledge they use for reasoning and decision-making.
In contrast, LLMs still struggle with knowledge integration after pre-training. While RAG has become a popular solution for long-term memory in LLMs, current methods struggle to integrate knowledge across different sources because each new document is encoded separately.
Many real-world applications require knowledge integration across multiple documents. For example, scientific literature reviews, legal case briefings, and medical diagnosis require connecting facts and concepts mentioned in different sources. Standard multi-hop question answering also requires integrating information between multiple passages of various documents.
To solve such tasks, current RAG systems use multiple retrieval and generation steps to join disparate passages. However, even perfectly executed multi-step RAG is often insufficient for many knowledge integration scenarios.
HippoRAG
HippoRAG takes inspiration from the biological interactions between the neocortex and the hippocampus that enable the powerful context-based, continually updating memory of the human brain.
According to scientific research, the neocortex processes and stores actual memory representations. The hippocampus holds a set of interconnected indices that point to memory units in the neocortex and store associations between them. The two components work together to accomplish “pattern separation,” which ensures that the representations of distinct experiences are unique, and “pattern completion,” which enables the retrieval of complete memories from partial stimuli.
HippoRAG mimics this memory model by using an LLM to transform a corpus of documents into a knowledge graph that acts as an artificial hippocampal index. HippoRAG works in two phases, offline indexing for storing information and online retrieval to integrate knowledge into user requests.
The offline indexing phase, analogous to memory encoding in the brain, uses an instruction-tuned LLM to extract important features from passages in the form of knowledge graph triples. This allows for more fine-grained pattern separation compared to dense embeddings used in classic RAG pipelines.
The artificial hippocampal index is an open knowledge graph built on the triples obtained from the retrieval corpus. To connect the components, HippoRAG uses off-the-shelf dense encoders fine-tuned for retrieval, which provide additional edges between similar noun phrases in the knowledge graph to aid in downstream pattern completion. The encoders help make the system more resilient and flexible as the same knowledge can be expressed in slightly different ways.
The online retrieval process mirrors the brain’s memory retrieval. The LLM extracts salient named entities from a user query, which it then links to nodes in the knowledge graph based on similarity determined by the retrieval encoders. HippoRAG then uses the Personalized PageRank algorithm, a version of PageRank that uses a set of user-defined source nodes to distribute probabilities across a graph. This constraint enables HippoRAG to limit its output to the set of candidate query nodes selected in the previous step. The results are then applied to the indexed passages to select them for retrieval.
HippoRAG in action
The researchers evaluated HippoRAG’s retrieval capabilities on two challenging multi-hop question answering benchmarks, MuSiQue and 2WikiMultiHopQA, and the HotpotQA dataset. They compared it against several strong retrieval methods and recent LLM-augmented baselines.
The results show that HippoRAG outperforms all other methods, including LLM-augmented baselines, on single-step retrieval. When combined with the multi-step retrieval method IRCoT, HippoRAG provides complementary gains of up to 20% on the same datasets.
A major advantage of HippoRAG is its ability to perform multi-hop retrieval in a single step. Its online retrieval process is also 10 to 30 times cheaper and 6 to 13 times faster than current iterative retrieval methods like IRCoT while achieving comparable performance.
“Our proposed neurobiologically principled methodology, although simple, already shows promise for overcoming the inherent limitations of standard RAG systems while retaining their advantages over parametric memory,” the researchers write. “HippoRAG’s knowledge integration capabilities, demonstrated by its strong results on path-following multi-hop QA and promise on path-finding multi-hop QA, as well as its dramatic efficiency improvements and continuously updating nature, makes it a powerful middle-ground framework between standard RAG methods and parametric memory and offers a compelling solution for long-term memory in LLMs.”
The researchers acknowledge several limitations that can be addressed to further improve HippoRAG, such as fine-tuning its components and further validating its scalability to much larger knowledge graphs. However, the current results already demonstrate the potential of neurobiologically-inspired methods to advance knowledge integration in LLMs and enable them to perform more complex reasoning over unstructured information.
The combination of knowledge graphs and LLMs is a very powerful tool that can unlock many applications. GNN-RAG, a recent method that combines graph neural networks (GNN) and knowledge graphs with LLMs has proven to solve many complex reasoning problems that were not possible with classic RAG. Adding GNNs to HippoRAG can be an interesting direction to explore in the future.