Saving and loading a DocumentSummaryIndex using a vector store

[mlevtov]

Hi,

I would like to persist a DocumentSummaryIndex instance in a vector store. At a later time, I would like to load the index from the vector store, retrieve summaries for existing nodes, and add new documents.

I have only been able to find documentation on persisting DocumentSummaryIndex to a file, not to a vector store.

Is this possible. If so, how?

Thanks

[dosubot[bot]]

Hey @mlevtov! Great to see you back. Hope you're doing awesome! 🚀

Based on the context provided, it seems that the LlamaIndex codebase does not directly support persisting a DocumentSummaryIndex instance to a vector store. The available persist method in the DocArrayInMemoryVectorStore class supports saving the state of the in-memory vector store to a file system, which can be used as a persistence layer. Here's how you can use it:

vector_store = DocArrayInMemoryVectorStore(index_path="your_index_path")
# Assuming you have your DocumentSummaryIndex instance loaded into vector_store
vector_store.persist("path_to_save_your_index")

However, loading the persisted index back into memory is not directly supported. You might need to extend the DocArrayInMemoryVectorStore class to add a loading mechanism or utilize the underlying docarray package's capabilities to load the persisted index.

After loading the index, you can use the methods provided by the docarray package or the DocArrayInMemoryVectorStore class to retrieve summaries for existing nodes and add new documents.

On the other hand, the SimpleVectorStore class provides a persist method that allows for persisting the state of the SimpleVectorStore, which includes embeddings and associated metadata, directly to a specified path on a filesystem or compatible storage system. This method serializes the SimpleVectorStoreData instance (which can be considered as a representation of a DocumentSummaryIndex in this context) to JSON and writes it to the specified path.

def persist(
    self,
    persist_path: str = os.path.join(DEFAULT_PERSIST_DIR, DEFAULT_PERSIST_FNAME),
    fs: Optional[fsspec.AbstractFileSystem] = None,
) -> None:
    """Persist the SimpleVectorStore to a directory."""
    fs = fs or self._fs
    dirpath = os.path.dirname(persist_path)
    if not fs.exists(dirpath):
        fs.makedirs(dirpath)

    with fs.open(persist_path, "w") as f:
        json.dump(self._data.to_dict(), f)

This method demonstrates the capability to persist the data of a SimpleVectorStore to a storage system, using a vector-like storage approach rather than traditional file system storage.

Please note that these methods are based on the current state of the LlamaIndex codebase and may not fully meet your requirements for persisting and loading a DocumentSummaryIndex instance in a vector store. You might need to adapt these methods or develop a custom solution based on your specific needs and the capabilities of the docarray package and the LlamaIndex codebase.

For more details, you can refer to the source code of the DocArrayInMemoryVectorStore and SimpleVectorStore classes in the LlamaIndex repository:

• DocArrayInMemoryVectorStore
• SimpleVectorStore

I hope this helps! If you have further questions or need more clarification, feel free to ask.

Sources

• llama-index-legacy/llama_index/legacy/vector_stores/docarray/in_memory.py
• llama-index-legacy/llama_index/legacy/vector_stores/simple.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[aijdsofttech]

I think loading summary index InMemoryVectorStore it's took huge ram so is there anyother option for storing summary index in vector store in pinecone@dosubot

[erlebach]

Given the huge number of methods and integrations already available, why isn't it possible to persist a DocumentSummaryIndex to a VectorStore? Given the cost of generating and using the index many times, I think avoiding recomputation would be a priority. It is also useful to perform experiments. If the index cannot be persisted, how might one write our own saving and loading mechanism for the DocumentSummaryIndex?

[alexandruvesa]

i'm also curious how a DocumentSummaryIndex can be persist to a VectorStore

[axhiao]

Is there any solution to this?

[yashsandansing]

I tried finding a solution for this but the base code for llama-index does not contain any feature to persist a DocumentSummaryIndex inside a vector store. It's mainly due to the different structures of a normal VectorStoreIndex and DocumentSummaryIndex.
What I ended up doing, in the end, was creating a customDocSummaryIndex class that worked the same as a DocumentSummaryIndex but in a VectorStoreIndex.
Most vector stores have an option for metadata that you can exploit to safely store your summaries.
My custom class created the document summaries using TreeSummarize and stored the summaries (page and document) inside the metadata of each page.

[axhiao]

Hi, you might want to try the code below, which worked for me. It will store the data in a vector database. Feel free to choose any vector database you prefer. I believe the key is to set up the storage context correctly.

vector_store_db = MilvusVectorStore(
        uri="http://localhost:19530", 
        collection_name='demo', 
        dim=1536, 
        overwrite=False
    )
storage_context = StorageContext.from_defaults(vector_store=vector_store_db)
doc_summary_index = DocumentSummaryIndex.from_documents(
    city_docs,
    llm=chatgpt,
    storage_context=storage_context, 
)

[axhiao]

Hi, apologies for the previous incorrect response. I believe I’ve found the right solution now, which you can try.

In llamaindex, the storage context consists of three main components: the Document Store, Index Store, and Vector Store. You can learn more about this in the documentation here. The Document Store and Index Store typically need to be stored in a standard database like MongoDB or Redis, while the Vector Store requires a vector database, such as Milvus.

So, if you want to store a DocumentSummaryStore, you’ll need to use a NoSQL database like Redis. You can find more details in this example.