fee based txs/chunks selector

[tsachiherman]

What ?

This PR adds support for selecting the largest set of transactions / chunks based on their fees.

In this PR, I haven't included any usage of that algorithm, but rather just the algorithm itself ( + unit test )

Description

LargestSet takes a slice of dimensions and a dimensions limit, and find the
largest set of dimensions that would fit within the provided limit. The return
slice is the list of indices relative to the provided [dimensions] slice.
note that the solution of the largest set is not
deterministic ( by the nature of the problem, there could be multiple "correct"
and different results )

Algorithm ( high level )

1. Scaling: Each dimension is scaled relative to its respective limit
   to ensure fair comparison. This step normalizes the dimensions,
   preventing larger dimensions from dominating the selection process.
2. Vector Sizing: Each dimension is treated as a vector, and its
   magnitude is calculated. This metric is used to assess the
   relative significance of each dimension.
3. Greedy Selection: Dimensions are iteratively selected based on
   their scaled magnitudes, adding them to the subset until the size
   limit is reached. The greedy approach prioritizes the most significant
   dimensions, maximizing the overall subset size.

Implementation notes:

• Precision: To mitigate potential precision issues arising from
  small-scale dimensions, the function employs a scaling factor to
  amplify values before normalization. This ensures accurate calculations
  even when dealing with very small numbers.
• Efficiency: The squared magnitude calculation is used as a proxy
  for Euclidean distance, optimizing performance without sacrificing
  accuracy for relative comparisons. This optimization avoids the
  computationally expensive square root operation.

Motivation

The motivation behind this algorithm, in contrast to a FIFO ordering is the following -

1. When using a FIFO ordering ( more precisely, an optimistic attempt to include the transactions in the order they were received ), we are effectively using a random ordering. While this random ordering could be the ideal one, it would rarely be such.
2. Any stable algorithm ( such as this ) would allow us to prioritize transaction volume throughput over chunk/block saturation.

Consider the following ( single dimension ) challenge:

1. You have 5 transactions with the following weights [5,5,1,1,1]. The chunk capacity is 4.
2. Using FIFO, you would generate the following chunks [5][5][1,1,1]
3. Using the above algorithm, you would have generate the following [1,1,1][5][5].

This doesn't seems very meaningful as is, but imagine that you keep getting a stream of 3 transaction of weight 1 once every chunk/block ( high demand scenario ). Now, you would be generating the following:

1. In the case of FIFO: [5],[5],[1,1,1,1,1],[1,1,1,1,1]..
2. In case of the above algorithm: [1,1,1][1,1,1],...,[5],[5]

Hence, the network would favor TPS over chunk/block saturation. Big transaction would get pushed down the road.

[aaronbuchwald]

Can we change the function signature to AddDimensions(a Dimensions, b Dimensions) since this is returning a new dimensions object rather than mutating to match the other functions in this package (ex. Add) ?

[aaronbuchwald]

Can we fix the typo AddDimensions as well?

[aaronbuchwald]

Can we add a benchmark for this function? I'm curious the performance difference of making this O(n^2) due to append remainder of the slice

[tsachiherman]

you're correct that it would create a O(n^2) in the worst case scenario. I'll address that..

[aaronbuchwald]

Where is this coming from?

[tsachiherman]

It creates arbitrarily set of dimensions, that are uniformly distributes and are (mostly) non-identical.
I believe that for the purpose of evaluation the overall performance of the algorithm, it would provide a reasonable test-set. We could (naturally) pick a different test set, but I wouldn't expect any material performance difference.

[aaronbuchwald]

Could we make this more obvious by using a random number modulo the desired upper bound rather than constructing it this way?

Could we also pre-generate if we're going to use the same seed and make sure to reset the timer when starting the actual benchmark?