Cumulative Sum, downsampling

[andre-merzky]

Hi all,

thanks for this module - we are really happy how datatables handles our reasonably large data sets! I do have two questions though:

• I have a column A of integers (positive and negative), and would like to add a column B with the commulative sum of those numbers. So, if A = [1, 2, 4, -4, 2], then I want to add B = [1, 3, 7, 3, 5]. How would I go about that?

• how would I downsample a column? Assume I have a table A with two columns of one million entries and want to derive a table B which has the same two columns with 1000 entries, where each entry is the average / max / mean / ... of the original table. I am not even sure if I am phrasing that question the right way: essentially I want to reduce the raw data to a size which can be plotted.

Many thanks for your time and effort!
Andre.

[st-pasha]

We don't have cumulative sum functionality yet, this needs to be implemented.

As for downsampling, there are 2 alternatives:

• the aggregate() function (https://datatable.readthedocs.io/en/latest/api/models/aggregate.html) which reduces a frame down to the specified number of "exemplars" (with counts). This is the function that we generally use for plotting.
• custom aggregation: if you want to split into nbins then create a column with group indices, and then run whatever groupby operation you want

nrows_per_bin = DT.nrows / nbins
DT['ix'] = dt.int32(dt.FExpr(range(DT.nrows)) / nrows_per_bin)
DT[:, dt.max(f[:]), by(f.ix)]

[andre-merzky]

Thank you, I'll give that a try!

[ghost]

Cumulative sum functionality would be a very welcome addition for me too.