initial gRPC spec changes for supporting index creation and querying #8829
Conversation
zehiko
added
exclude from changelog
|
Latest documentation preview deployed successfully.
Note: This comment is updated whenever you push a commit. |
|
Web viewer built successfully. If applicable, you should also test it:
Note: This comment is updated whenever you push a commit. |
| @@ -9,6 +9,16 @@ service StorageNode { | |||
| rpc Query(QueryRequest) returns (stream DataframePart) {} | |||
| rpc FetchRecording(FetchRecordingRequest) returns (stream rerun.common.v0.RerunChunk) {} | |||
|
|
|||
| rpc IndexCollection(IndexCollectionRequest) returns (IndexCollectionResponse) {} | |||
There was a problem hiding this comment.
Right now everything lives under one service StorageNode. I wonder if it is cleaner to create several services that groups similar calls logically? Maybe something like CatalogService, IndexService, ...? It might be nice in the future to decide on a fine granular level which services should spin up, for example if we want to distribute loads across multiple VMs?
There was a problem hiding this comment.
there is still a tight relation between catalog, collection, collection query path, collection index query path. This definitely requires bit of thinking how we split it. I'll create an issue, but I won't tackle this as part of this story.
| @@ -9,6 +9,16 @@ service StorageNode { | |||
| rpc Query(QueryRequest) returns (stream DataframePart) {} | |||
| rpc FetchRecording(FetchRecordingRequest) returns (stream rerun.common.v0.RerunChunk) {} | |||
|
|
|||
| rpc IndexCollection(IndexCollectionRequest) returns (IndexCollectionResponse) {} | |||
There was a problem hiding this comment.
In addition to CreateIndex, we probably want an API for ReIndex that only requires the collection + ColumnDescriptor but doesn't need the parameters.
There was a problem hiding this comment.
Tracking which recordings are in the index so we know whether reindexing is necessary would be nice to think about, but not necessary yet.
There was a problem hiding this comment.
https://github.com/rerun-io/dataplatform/issues/156
This will be tackled in a first follow up.
zehiko
changed the title
…le with IndedColumn
This reverts commit 8ab34a8.
What
Expose creating and querying an index over an entire collection of data.
We add support for:
Query data is always represented with a record batch with just 1 column that carries the query data.