Improve performance for large loads with dask

[aulemahal]

The transparent use of dask within xarray is really nice, but it has the side effect of creating a dask task for every operation. In the case of xclim, with all those rolling and resample the number of tasks created is ridiculously high.

With large datasets, the scheduler is sometimes so overloaded that it never even begins the computation. Sometimes it crashes, sometimes it only hangs. Most of the time we get several "WARNING - full garbage collections...." and other warnings.

Rechunking to larger chunk sometimes helps and sometime is insufficient.

We had this problem in sdba and it was solved by wrapping every "unit" operation with map_blocks. This way, we combine many small operations in a single dask task. However, this solution has a lot of drawbacks. It's complicated to maintain, has many bugs with "auxiliary" coords and it's just hard to read.

The idea is there though. Could we implement something that would wrap the compute into a single dask task, from within the indicator's __call__? I guess it should be controlled by an option, and maybe only apply to indicators performing resampling, since those are the main victim of task decuplation?

[aulemahal]

This issue is out-of-scope for xclim. flox has improved the "resample" case and the "rolling" one might have to be improved by a similar process : within xarray or a plugin like flox.

[dcherian]

It'd be good to see a minimal example of the rolling problem. Also pydata/xarray#7344 (comment)