I don’t able to complete this scrapping challenge

palashrai31 · January 30, 2021, 2:49pm

Please Provide the right path and complete tutorial of this challenge.

prashant_ml · January 30, 2021, 3:39pm

hey @palashrai31 ,
The hints are provided in the video itself.
But still ,
you can tell me at what particular things you are stuck at , i will surely help you out of it.

palashrai31 · January 30, 2021, 3:46pm

Yeah sure, I m unable to iterate over the URLs of a different item of the same product and fetching its details.
I m getting none in the URL list
It’s been very confusing and I m trying it from last 2 days but still not getting the required output

palashrai31 · January 30, 2021, 3:48pm

can you plz provide the working code for the better understanding

prashant_ml · January 30, 2021, 3:53pm

hey @palashrai31,
I am sorry but currently i don’t have any such code for this.
You can share me yours , through github or drive link , i will check into it and let you know.

palashrai31 · January 30, 2021, 4:00pm

okay I will check it one more time and then will send it to you by tomorrow

prashant_ml · January 30, 2021, 4:01pm

yeah sure , No problem buddy.

palashrai31 · January 31, 2021, 2:18pm

github.com

palash3112/pepperfry/blob/master/pepperfry_challenge/pepperfry_challenge/spiders/pepperfry_.py

import scrapy
import requests
import json
import os

class pepperfry(scrapy.Spider):
    name = "pepperfry"
    BASE_DIR='./pepperfry/'
    Max_Cnt=0  

    def start_requests(self):
        BASE_url ="https://www.pepperfry.com/site_product/search?q="
                  
       
        items=['arm chairs','two seater sofa','bean bags','bench','book case','chest drawers','coffee tables','dining set',
              'king beds','queens bed','garden seating']
        urls_l=[]
        dir_names=[]
        for item in items:
            url_string="+".join(item.split(' '))

This file has been truncated. show original

palashrai31 · January 31, 2021, 2:20pm

i tried to run the code given in hit video but not getting proper output and also getting DEBUG: crawled (200) message

palashrai31 · January 31, 2021, 2:24pm

link : https://github.com/palash3112/pepperfry

prashant_ml · February 1, 2021, 6:26am

hey @palashrai31 ,
first you need to make change in is that , you need to use the same name as of your project and name variable inside the spider .
currently its pepperfry change it to pepperfry_challenge.
It will work. Now coming to its functioning , there is still some error in functioning , and now i would request you to kindly look into and try.

Keep in mind while making nested crawling , the url you provide them and how do you work on them , they will matter a lot.

prashant_ml · February 10, 2021, 7:38am

I hope I’ve cleared your doubt. I ask you to please rate your experience here
Your feedback is very important. It helps us improve our platform and hence provide you
the learning experience you deserve.

On the off chance, you still have some questions or not find the answers satisfactory, you may reopen
the doubt.