Merge the vocexplorer into vocdoni-node

[p4u]

Since vocdoni-node includes badgerhold kv database, the scrutinizer is being converted to a new component named indexer. This renaming is still pending, but the code is already been modified.

There is a new database (different from the blockchain state one) defined here that might be extended: https://github.com/vocdoni/vocdoni-node/blob/master/vochain/scrutinizer/db.go

New calls are already defined and many others might be defined as can be seen on this code snippet: https://github.com/vocdoni/vocdoni-node/blob/master/vochain/scrutinizer/process.go#L59

So the list of steps would be:

1. Identify all required API query calls that the current explorer needs. Iterate on them and try to keep them as generic and useful as possible, so other components might also use it. The resulting API will be part of the "Gateway" and properly documented on https://docs.vocdoni.io (currently named "results API" but it will change to "indexer API".
2. Rename scrutinizer to indexer in vocdoni-node
3. Extend the database and the query methods in order to support all queries required by the explorer
4. Use the explorer frontend with the vocdoni-node indexer as backend

With this, we'll have a single basecode for the Vochain explorer and the Vocdoni node (so no more pain on the vocexplorer side when upgrading vocdoni-node module). And we will have a powerful API on the Gateway that will be consumed by the Explorer but it can also be used for third-party integrations.

[p4u]

@NateWilliams2 @emmdim @jordipainan

[NateWilliams2]

I like this proposal, and I can start compiling a list of current and desired apis for the explorer.

When you say “a single base code for the Vochain explorer and the Vocdoni node”, you’re referring to the explorer backend only, yes? I think the explorer itself should be hosted separately for the node. I’d propose to create a new repo specifically for the explorer frontend. Possibly to start this could simply be a fork of the current explorer frontend with updated apis, and a simple server for hosting the html + wasm file. And in the mid-long term this could be refactored into react or flutter web, etc.

[p4u]

Great!

So lets analyze this list.

1. All Process and Entity related queries are already covered by the current scrutinizer but the Search ones, that might be added (should be easy).
2. The Envelope list is currently not stored by the scrutinizer in order to minimize the storage (having envelopes duplicated in state and scrutinizer would be a big pain). So I'd try to use the state database and write the helper functions to scrutinizer/indexer (its mainly porting what you already have).
3. The Validators is a very small dataset, so should be fine if we just use the state. I don't see the need for duplicating it in a more indexable way either.
4. The Transaction and the Blocks are the two big missing on the current scrutinizer. But I think we might just add them.

[p4u]

So if you want you can start adding the missing methods to the current scrutinizer
https://github.com/vocdoni/vocdoni-node/tree/master/vochain/scrutinizer

[NateWilliams2]

@p4u I've run into the first issue, which is that I seem to have no way of querying for a list of envelopes without first knowing their process ids. This means there's no way to get a list of all envelopes by their creation time.
I have a few ideas:
A) Re-design the front-end so that envelopes are only displayed per process, and never in a list all together
B) Index envelope nullifiers in the VochainState by some global envelope height (or some other indexable)
C) Store envelopes in the scrutinizer db, so that they can easily be indexed with various keys

What do you think?

[NateWilliams2]

Based on our conversation today, it seems like either A) or C) (but modified to index only nullifiers and not entire envelopes) will work.
I think we'll need to implement a mix of the two. It seems fine to remove the global envelope list, as this doesn't really make sense from a use perspective. But in order for users to be able to search for their envelopes by nullifier, we still need to have some other index. Currently the State.Envelope method requires processId and Nullifier. I think requiring users to first select a process and then search for their vote envelope is too restrictive.

this specific problem with searching could be fixed by indexing nullifier->processId in the indexer. So the flow would be

• search "0xabc123..."
• find {"0xabc12345...", processId} (indexer)
• find envelope itself (state)

[p4u]

I think we might add a new database Struct here https://github.com/vocdoni/vocdoni-node/blob/master/vochain/scrutinizer/db.go

Let's try to store only the minimum required information. I'd propose something like:

type Envelope {
 Nullifier HexBytes => primary key
 ProcessId HexBytes => index
 Timestamp time.Time
}

Does it full fill your needs?

[p4u]

We can consider this done :)