add length to progress bar

[xsuchy]

so the progress bar shows real progress

Fixes #3342

Tasks

• Reviewed contribution guidelines
• PR is descriptively titled 📑 and links the original issue above 🔗
• Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
  Run tests locally to check for errors.
  The testsuite is broken for me and produce lots of not relevant errors. I will appreciate testing.
• Commits are in uniquely-named feature branch and has no merge conflicts 📁
• [-] Updated documentation pages (if applicable)
• Updated CHANGELOG.rst (if applicable)

[pombredanne]

Miroslav and thanks!
There is an issue wrt. to making this work with large codebase. Even a linux kernel tarball may be problematic... we may need to find a better way.

[pombredanne]

This will be a problem when we have a codebase with 10 million files, we may need to find a better way? ... otherwise the system will literally be stalled for minutes before displaying a progress bar.

[xsuchy]

For the kernel, the delay is 45 sec until it traverses 200k files. While the scan itself runs several hours.
With -q there is no delay.

[pombredanne]

Why the times 2?

[xsuchy]

Because #3344 Otherwise the progressbar go to 170/170 and then continue to 340/170. This *2 can be removed when we find when we find where is the count done twice. I have a suspicion, but it needs more testing.

[xsuchy]

Several things I considered as an option:

1. instead resources = list(((r.location, r.path) for r in codebase.walk() if r.is_file)) use simpler resources = list((1 for r in codebase.walk() if r.is_file)) because we just need the count. This is consume less memory, but does not give any measurable time.

2. pass the resources to scan_codebase so we do not need to count it twice. Hmm. I will do this. PR updated.

[xsuchy]

I tried to improve the performance of walk() in aboutcode-org/commoncode#78