URLHeadBear.py: Use robots.txt #2891
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Requests that are not allowed by robots.txt are reported. Closes coala#1782
|
|
|
Tests have not been added. I wanted to know if this approach is okay before modifying existing tests. |
frextrite
reviewed
Mar 18, 2019
There was a problem hiding this comment.
Changes look good. You can go ahead and add tests now.
But I have one suggestion. Do we really need to show a message to the user if the robots.txt doesn't allow crawling a specific sub-directory? Since it would give a false belief to the user that the mentioned link doesn't work and should be changed, whereas it actually meant that no crawlers are allowed to visit the specific page.
|
@frextrite The message has to be shown. The issue asks the situation to be reported. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Requests that are not allowed by robots.txt
are reported.
Closes #1782
For short term contributors: we understand that getting your commits well
defined like we require is a hard task and takes some learning. If you
look to help without wanting to contribute long term there's no need
for you to learn this. Just drop us a message and we'll take care of brushing
up your stuff for merge!
Checklist
them.
individually. It is not sufficient to have "fixup commits" on your PR,
our bot will still report the issues for the previous commit.) You will
likely receive a lot of bot comments and build failures if coala does not
pass on every single commit!
After you submit your pull request, DO NOT click the 'Update Branch' button.
When asked for a rebase, consult coala.io/rebase
instead.
Please consider helping us by reviewing other peoples pull requests as well:
corobo mark wip <URL>to get it outof the review queue.
The more you review, the more your score will grow at coala.io and we will
review your PRs faster!