Unable to save webpage as html

SanchitSayala · April 16, 2020, 9:54am

I wrote the code shown in the video to externally save the webpage as html file. But it’s not working for me.

SanchitSayala · April 16, 2020, 9:55am

The termilal shows:

S18CRX0120 · April 16, 2020, 2:45pm

Hey @SanchitSayala, copy past code from here, and check if this works or not,

coding-blocks-archives/machine-learning-online-2018/blob/master/Web Scrapping/Scrapy Tutorial/myproject/myproject/spiders/quotes_spider.py

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"

    def start_requests(self):
        urls = [
            'http://quotes.toscrape.com/page/1/',
        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        page = response.url.split("/")[-2]
        filename = 'quotes-%s.html' % page
        
        for q in response.css("div.quote"):
            text = q.css('span.text::text').get()
            author = q.css('small.author::text').get()

This file has been truncated. show original

If this works, than check your code line by line from this.

Hope this resolved your doubt.
Plz mark it as resolved in my doubts section.

lousybrick · June 9, 2020, 6:33am

When you run ‘scrapy crawl’ on cmd make sure you type the name you’ve given your spider inside your quotes_spider.py file instead of typing the python file name. For Example:

My .py file is saved as quotes_spider.py

class QuotesSpider(scrapy.Spider):
name = "quotes"

So on cmd type :
‘scrapy crawl quotes’ (this is the name you’ve given your spider inside your QuotesSpider class)
Instead of ‘scrapy crawl quotes_spider’ (which is the .py file name)