Add tests for Scrapy Crawlers

[Anmolbansal1]

Expected Behavior

There should be a test method to test all scrapy crawlers for -

• consistent output across all crawlers
• proper hyperlinks in pages
• all content is extracted from news page
• they should be lite such that they can be enforced with travis ci

Current Behavior

Currently there are no tests.

Context

Tests makes the coding process more Agile. When more and more crawlers are added to scrapy project, sometimes structure needs to be changed. However, changing already-tested code is both risky and costly. If we have unit tests in place, then we can proceed for refactoring confidently.

Reference - Scrapy Contracts

[carpecodeum]

can i work on this?

[Kunal614]

can i work on this ?

[Anmolbansal1]

Yes sure!!

[Anmolbansal1]

@Kunal614 any update??

[Kunal614]

@Anmolbansal1
my mid-sem was going on , so now i will continue , can you explain what to do.

[Anmolbansal1]

@Kunal614 You need set up structure to test scrapy crawlers and ensure that they provide data in proper format and none of the columns are missing. Also integerate them to Travis CI.