Cache for HF file system is broken

[shcheklein]

Query to reproduce:

from datachain import DataChain, C

from datachain.lib.tar import process_tar

(
  DataChain.from_storage("hf://datasets/mozilla-foundation/common_voice_17_0")
    .settings(parallel=4, cache=True, prefetch=4)
  	.filter(C("file.path").glob("*/en/train/*.tar"))
  	.gen(file=process_tar)
    .save("common-voice-index")
)

[shcheklein]

#747

Related #746

[shcheklein]

@skshetry can this be closed?

[skshetry]

Cache is still broken. The above script might work because #730 stopped prefetching for HF.

[shcheklein]

@skshetry thanks! does require an upstream fix? could you please summarize (or copy from the previous tickets / discussion).

[skshetry]

The base Client implementation expects an async filesystem. PTAL at scandir and other async methods in Client.

HF filesystem does not offer an asynchronous API, so there are two issues in our implementation:

1. some fsspec methods that Client expects does not exist (eg: _info, etc).

2. We have some async APIs implemented in HFClient that calls synchronous function.

   datachain/src/datachain/client/hf.py

   Lines 34 to 35 in 3767173

   	async def ls_dir(self, path):
   	return self.fs.ls(path, detail=True)

   But this is a bad thing to do, as it blocks the event loop. (Similar stuff is done in FileClient, and I am surprised that this thing even passed reviews).

Luckily for caching, it seems we are using client.download and client.put_in_cache which are both synchronous function, so we could reimplement them in HFClient. And that would fix usages from File API.

For usages in AsyncMapper and other places where we need to concurrently perform operations, unfortunately, there are no good solutions. Either we'll be blocking event-loop by running a synchronous function, or we'll need to run the operation in another thread.

fsspec can do the latter automatically, it can generate AsyncFileSystem from a synchronous filesystem, where the operations are run in another thread.