[Question]: Does that merge storage is posibble?

[284nnuS]

Question Validation

• I have searched both the documentation and discord for an answer.

Question

Hello, i try to add mutiple storage with this code :

index_private = load_indices_from_storage(
                storage_context=StorageContext.from_defaults(persist_dir=os.environ['root'][:-4]+'/kb/'+project_id)
            )
            index_public = load_indices_from_storage(
                storage_context=StorageContext.from_defaults(persist_dir=os.environ['root'][:-4]+'/kb/public')
            )

            index_private.append(index_public)

            index = index_private[0]
            print(len(index))
            query_engine = index.as_query_engine(
                text_qa_template=self.qa_template,
                similarity_top_k=1,
            )

 But it seem be not work. So does that merge storage is posible

[dosubot[bot]]

🤖

Hi there,

From your code, it seems like you're trying to merge two indices, index_private and index_public. However, the append() method you're using is not the correct way to merge these indices in LlamaIndex.

Could you please provide more information about the error message you're receiving? Also, could you clarify what you're trying to achieve by merging these two indices? This will help me understand your issue better and provide a more accurate solution.

Thanks for your cooperation.

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[284nnuS]

🤖

Hi there,

From your code, it seems like you're trying to merge two indices, index_private and index_public. However, the append() method you're using is not the correct way to merge these indices in LlamaIndex.

Could you please provide more information about the error message you're receiving? Also, could you clarify what you're trying to achieve by merging these two indices? This will help me understand your issue better and provide a more accurate solution.

Thanks for your cooperation.

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

So how can I merge two indices from 2 directory? Could you give me some document

[dosubot[bot]]

🤖

Hello,

From the context provided, it seems that the append() method you're trying to use doesn't exist in the LlamaIndex framework. The method that is used to insert nodes into the index graph is insert().

Here is the relevant code:

def insert(self, nodes: Sequence[BaseNode]) -> None:
    """Insert into index_graph."""
    for node in nodes:
        self._insert_node(node)

The _insert_node() method handles the logic of where to insert the node in the index graph. If the graph is empty, the node is inserted as a root node. If the current graph nodes are leaf nodes, the node is inserted under the parent node. If the current graph nodes are not leaf nodes, the method tries to find the right summary node to insert under. If a number cannot be extracted or if the number is out of range, the node is inserted under the parent node. If a valid number is extracted, the node is inserted under the selected node.

After the node is inserted, the method updates the summary for the parent node, if it exists. The updated summaries are then bubbled up the tree. The summary is updated by truncating the text chunks of the current graph nodes and predicting a new summary using the llm_predictor. The new summary is then set as the content of the parent node.

You can find this in the llama_index/indices/tree/inserter.py file.

To merge your two indices, you might need to extract the nodes from index_public and then insert them into index_private using the insert() method. However, without more information about the structure of your indices and the BaseNode class, it's hard to provide a more specific solution.

I hope this helps! If you have more questions or need further clarification, feel free to ask.

Sources

• llama_index/indices/tree/inserter.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[284nnuS]

🤖

Hello,

From the context provided, it seems that the append() method you're trying to use doesn't exist in the LlamaIndex framework. The method that is used to insert nodes into the index graph is insert().

Here is the relevant code:

def insert(self, nodes: Sequence[BaseNode]) -> None:
    """Insert into index_graph."""
    for node in nodes:
        self._insert_node(node)

The _insert_node() method handles the logic of where to insert the node in the index graph. If the graph is empty, the node is inserted as a root node. If the current graph nodes are leaf nodes, the node is inserted under the parent node. If the current graph nodes are not leaf nodes, the method tries to find the right summary node to insert under. If a number cannot be extracted or if the number is out of range, the node is inserted under the parent node. If a valid number is extracted, the node is inserted under the selected node.

After the node is inserted, the method updates the summary for the parent node, if it exists. The updated summaries are then bubbled up the tree. The summary is updated by truncating the text chunks of the current graph nodes and predicting a new summary using the llm_predictor. The new summary is then set as the content of the parent node.

You can find this in the llama_index/indices/tree/inserter.py file.

To merge your two indices, you might need to extract the nodes from index_public and then insert them into index_private using the insert() method. However, without more information about the structure of your indices and the BaseNode class, it's hard to provide a more specific solution.

I hope this helps! If you have more questions or need further clarification, feel free to ask.

Sources

* [llama_index/indices/tree/inserter.py](https://github.com/jerryjliu/llama_index/blob/-/llama_index/indices/tree/inserter.py)

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

Could you provide me a document for extracting node form vector store index

[dosubot[bot]]

🤖

Hello,

To merge two indices in the LlamaIndex framework, you indeed need to extract nodes from one index and insert them into another. However, the LlamaIndex framework doesn't provide a direct method to extract nodes from an index. Instead, you can use the query method of the MongoDBAtlasVectorSearch class to find the most similar nodes to a given query, which can be used to extract nodes from an index.

Here is a sample code snippet on how you can use the query method:

from llama_index.vector_stores.types import VectorStoreQuery

# Initialize the MongoDBAtlasVectorSearch object
mongodb_vector_search = MongoDBAtlasVectorSearch(
    mongodb_client=<your_mongodb_client>,
    db_name=<your_db_name>,
    collection_name=<your_collection_name>,
    index_name=<your_index_name>
)

# Create a VectorStoreQuery object
query = VectorStoreQuery(
    query_embedding=<your_query_embedding>,
    similarity_top_k=<your_similarity_top_k>,
    filters=<your_filters>
)

# Query the index for the top k most similar nodes
query_result = mongodb_vector_search.query(query)

# Extract the nodes from the query result
nodes = query_result.nodes

In this code snippet, <your_mongodb_client>, <your_db_name>, <your_collection_name>, <your_index_name>, <your_query_embedding>, <your_similarity_top_k>, and <your_filters> should be replaced with your actual MongoDB client, database name, collection name, index name, query embedding, top k value, and filters respectively.

Once you have extracted the nodes from index_public, you can insert them into index_private using the insert() method. The insert() method in the LlamaIndex framework is used to insert nodes into the index graph. It takes a sequence of nodes as an argument and inserts each node into the graph.

Here is the relevant code:

def insert(self, nodes: Sequence[BaseNode]) -> None:
    """Insert into index_graph."""
    for node in nodes:
        self._insert_node(node)

Please replace nodes with the nodes you extracted from index_public.

I hope this helps! If you have any other questions, feel free to ask.

Best,
Dosu

Sources

• llama_index/vector_stores/mongodb.py
• llama_index/indices/tree/inserter.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.