Scrape a book challenge using scrapy

#1

my code is only able to scrape first two webpages , what is the problem

the code is :

import scrapy
class bookscraper(scrapy.Spider):
name=“bookscrape”
start_urls=[“http://books.toscrape.com/”
]
def parse(self,response):
for entity in response.css(‘article.product_pod’):
image_address=entity.css(‘div.image_container img::attr(src)’).get()
title=entity.css(‘h3 a::attr(title)’).get()
price=entity.css(‘p.price_color::text’).get()

		yield {'Url':image_address,'Title':title,'Price':price
		}

	next_page = 'http://books.toscrape.com/'+ response.css('li.next a::attr(href)').get()
	if next_page is not None:
		yield response.follow(next_page, callback=self.parse)
#2

Hey,
the start url should be : http://books.toscrape.com/”catalogue/page-1.html
and next_page = response.css('li.next a::attr(href)').get()
Try with this.

Thanks :slight_smile:

#3

i have done this with scrapy and want to implement with beautiful soup , i made a scraper for one page and want to do this for multiple pages

#4

beautifulsoup alone will not let you do this.
You would also need selenium to do this automation ask.
Clicking and going to next page can be done with selenium library, try exploring that

#5

thanks , but its been 13 days when i asked this doubt , could you please solve doubts faster.
Its of no use if doubt solving takes this much time.

#6

I hope I’ve cleared your doubt. I ask you to please rate your experience here
Your feedback is very important. It helps us improve our platform and hence provide you
the learning experience you deserve.

On the off chance, you still have some questions or not find the answers satisfactory, you may reopen
the doubt.