Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crash downloading large library #73

Open
Oryhara opened this issue Mar 17, 2023 · 5 comments
Open

crash downloading large library #73

Oryhara opened this issue Mar 17, 2023 · 5 comments

Comments

@Oryhara
Copy link

Oryhara commented Mar 17, 2023

I bought this bundle: https://itch.io/bundle/download/fCCJhYMFZALEAiZP7yTQ6UTHjm7WHfNdoGrsnxez
and now i'm trying to download them all locally.
Installed this app per instructions. running command as indicated in readme.

It repeatedly gives this error:

Downloaded Clerics of Creation: Apocalypse and Genesis Domains (25 of 505)
Traceback (most recent call last):
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\site-packages\requests-2.28.1-py3.11.egg\requests\models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\json_init_.py", line 346, in loads
return _default_decoder.decode(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 198, in run_module_as_main
File "", line 88, in run_code
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\site-packages\itchiodl\downloader_main
.py", line 66, in
main()
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\site-packages\itchiodl\downloader_main
.py", line 62, in main
lib.download_library(args.platform)
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\site-packages\itchiodl\library.py", line 108, in download_library
for _ in r:
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\concurrent\futures_base.py", line 619, in result_iterator
yield _result_or_cancel(fs.pop())
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\concurrent\futures_base.py", line 317, in _result_or_cancel
return fut.result(timeout)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\concurrent\futures_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\concurrent\futures_base.py", line 401, in __get_result
raise self._exception
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\concurrent\futures\thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\site-packages\itchiodl\library.py", line 99, in dl
g.download(self.login, platform)
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\site-packages\itchiodl\game.py", line 75, in download
self.do_download(d, token)
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\site-packages\itchiodl\game.py", line 136, in do_download
j = r.json()
^^^^^^^^
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\site-packages\requests-2.28.1-py3.11.egg\requests\models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I'm having limited success running it repeatedly with thousands of jobs. But of the 254 creators in this bundle i've only gotten 68 downloaded so far.

@Emersont1
Copy link
Owner

Emersont1 commented Mar 17, 2023 via email

@Oryhara
Copy link
Author

Oryhara commented Mar 20, 2023

Idk what a cloud box is. I'm running this locally on a laptop using powershell.
I'm not sure if rate limiting makes sense because I had more success when I added a really high number of jobs.
and initially i ran with just 1 job.

@locketshoru
Copy link

locketshoru commented Mar 22, 2023

I'm having the same issue on win 10 LTSC, with the added issue of "I have no idea where the downloads are going, and so cannot check if they're actually working". ^^; A fix would be good, or at least instructions for someone who doesn't work with python very much. (I uh. Also don't know how to change the amount of jobs. ^^; )

@Emersont1
Copy link
Owner

I'm having the same issue on win 10 LTSC, with the added issue of "I have no idea where the downloads are going, and so cannot check if they're actually working"

Whatever directory the shell is currently in

@vampeyerate
Copy link

i am having the very same issue with the same bundle, but on a different game. mine stopped working after seyvoth manor. i know everything up till that point has downloaded, but after that it just freezes up and stops. here's the message it gives me:

Traceback (most recent call last):
File "", line 198, in run_module_as_main
File "", line 88, in run_code
File "C:\Users\pickl\AppData\Local\Programs\Python\Python312\Lib\site-packages\itchiodl\downloader_main
.py", line 66, in
main()
File "C:\Users\pickl\AppData\Local\Programs\Python\Python312\Lib\site-packages\itchiodl\downloader_main
.py", line 62, in main
lib.download_library(args.platform)
File "C:\Users\pickl\AppData\Local\Programs\Python\Python312\Lib\site-packages\itchiodl\library.py", line 108, in download_library
for _ in r:
File "C:\Users\pickl\AppData\Local\Programs\Python\Python312\Lib\concurrent\futures_base.py", line 619, in result_iterator
yield _result_or_cancel(fs.pop())
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\pickl\AppData\Local\Programs\Python\Python312\Lib\concurrent\futures_base.py", line 317, in _result_or_cancel
return fut.result(timeout)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\pickl\AppData\Local\Programs\Python\Python312\Lib\concurrent\futures_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\pickl\AppData\Local\Programs\Python\Python312\Lib\concurrent\futures_base.py", line 401, in __get_result
raise self._exception
File "C:\Users\pickl\AppData\Local\Programs\Python\Python312\Lib\concurrent\futures\thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\pickl\AppData\Local\Programs\Python\Python312\Lib\site-packages\itchiodl\library.py", line 99, in dl
g.download(self.login, platform)
File "C:\Users\pickl\AppData\Local\Programs\Python\Python312\Lib\site-packages\itchiodl\game.py", line 75, in download
self.do_download(d, token)
File "C:\Users\pickl\AppData\Local\Programs\Python\Python312\Lib\site-packages\itchiodl\game.py", line 187, in do_download
if utils.md5sum(out_file) != d["md5_hash"]:
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\pickl\AppData\Local\Programs\Python\Python312\Lib\site-packages\itchiodl\utils.py", line 51, in md5sum
with path.open("rb") as f:
^^^^^^^^^^^^^^^
File "C:\Users\pickl\AppData\Local\Programs\Python\Python312\Lib\pathlib.py", line 1013, in open
return io.open(self, mode, buffering, encoding, errors, newline)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'ladyorthetiger\birb-heist\Birb Heist — For Everyone.pdf'

that last filenotfound is actually one of the files that successfully downloaded. also another thing that may or may not be relevant is this bundle is like all pdfs and images, as it's a ttrpg bundle.
I'm not excellent at interpreting the errors it's throwing, any advice?
im on a laptop with windows 11 running this in the normal command prompt
thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants