Unable to scrape files through Spider

yashagarwal918 · April 20, 2020, 4:40pm

I tried scraping quotes from the page but when I run the command scrapy crawl spidername , the terminal shows Crawled 0 pages , resulting in closing the spider session.

S18CRX0120 · April 20, 2020, 7:15pm

Hey @yashagarwal918, copy and paste this code and than try it, if it runs successfully than match your code line by line from this,

github.com

coding-blocks-archives/machine-learning-online-2018/blob/master/Web Scrapping/Scrapy Tutorial/myproject/myproject/spiders/quotes_spider.py

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"

    def start_requests(self):
        urls = [
            'http://quotes.toscrape.com/page/1/',
        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        page = response.url.split("/")[-2]
        filename = 'quotes-%s.html' % page
        
        for q in response.css("div.quote"):
            text = q.css('span.text::text').get()
            author = q.css('small.author::text').get()

This file has been truncated. show original

Hope this resolved your doubt.
Plz mark the doubt as resolved in my doubts section.