Unable to create the Folders using scrapy

#1

I am having an error in creating the two folders displaying the html of the webpages to be scrapped. I think I am having an error in executing the required statement correctly in command prompt.

Here is the result after running the statement

Windows PowerShell
Copyright © Microsoft Corporation. All rights reserved.

Try the new cross-platform PowerShell https://aka.ms/pscore6

PS C:\Users\sanch> cd ‘.\Documents\Coding Blocks data science\Scrapy Tuto
rial\myproject’
PS C:\Users\sanch\Documents\Coding Blocks data science\Scrapy Tutorial\my
project> scrapy crawl quotes_spider
2020-04-12 02:34:33 [scrapy.utils.log] INFO: Scrapy 2.0.1 started (bot: m
yproject)
2020-04-12 02:34:33 [scrapy.utils.log] INFO: Versions: lxml 4.4.1.0, libx
ml2 2.9.9, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 20.3.0, P
ython 3.7.4 (default, Aug 9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)],
pyOpenSSL 19.0.0 (OpenSSL 1.1.1d 10 Sep 2019), cryptography 2.7, Platfo
rm Windows-10-10.0.18362-SP0
2020-04-12 02:34:33 [scrapy.utils.log] DEBUG: Using reactor: twisted.inte
rnet.selectreactor.SelectReactor
2020-04-12 02:34:33 [scrapy.crawler] INFO: Overridden settings:
{‘BOT_NAME’: ‘myproject’,
‘NEWSPIDER_MODULE’: ‘myproject.spiders’,
‘ROBOTSTXT_OBEY’: True,
‘SPIDER_MODULES’: [‘myproject.spiders’]}
2020-04-12 02:34:33 [scrapy.extensions.telnet] INFO: Telnet Password: d7b
e7eae77f75da5
2020-04-12 02:34:33 [scrapy.middleware] INFO: Enabled extensions:
[‘scrapy.extensions.corestats.CoreStats’,
‘scrapy.extensions.telnet.TelnetConsole’,
‘scrapy.extensions.logstats.LogStats’]
2020-04-12 02:34:33 [scrapy.middleware] INFO: Enabled downloader middlewa
res:
[‘scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware’,
‘scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware’,
‘scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware’
,
‘scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware’,
‘scrapy.downloadermiddlewares.useragent.UserAgentMiddleware’,
/min), scraped 0 items (at 0 items/min)
2020-04-12 02:34:33 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-04-12 02:34:33 [scrapy.core.engine] INFO: Closing spider (finished)
2020-04-12 02:34:33 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{‘elapsed_time_seconds’: 0.021613,
‘finish_reason’: ‘finished’,
‘finish_time’: datetime.datetime(2020, 4, 11, 21, 4, 33, 930013),
‘log_count/INFO’: 10,
‘start_time’: datetime.datetime(2020, 4, 11, 21, 4, 33, 908400)}
2020-04-12 02:34:33 [scrapy.core.engine] INFO: Spider closed (finished)

#2

Hey @sanchit123manchanda, your spider is working i guess, it has opened and closed accordingly. What’s the error you are facing ?

#3

The two quotes file having html code are not created.

#4

Hey @sanchit123manchanda, share your quotes_spider.py file by saving it on cb.lk/ide

#5

Now its showing a different error after I tried to do it again-

PS C:\Users\sanch> cd ‘.\Documents\Coding Blocks data science\Scrapy Tutorial\myproject’
PS C:\Users\sanch\Documents\Coding Blocks data science\Scrapy Tutorial\myproject> scrapy crawl quotes_spider
2020-04-12 22:07:51 [scrapy.utils.log] INFO: Scrapy 2.0.1 started (bot: myproject)
2020-04-12 22:07:51 [scrapy.utils.log] INFO: Versions: lxml 4.4.1.0, libxml2 2.9.9, cssselect 1.1.0, parsel 1.5.2, w3lib 1.
21.0, Twisted 20.3.0, Python 3.7.4 (default, Aug 9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)], pyOpenSSL 19.0.0 (OpenSSL
1.1.1d 10 Sep 2019), cryptography 2.7, Platform Windows-10-10.0.18362-SP0
2020-04-12 22:07:51 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
Traceback (most recent call last):
File “c:\users\sanch\anaconda3\lib\site-packages\scrapy\spiderloader.py”, line 68, in load
return self._spiders[spider_name]
KeyError: ‘quotes_spider’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “c:\users\sanch\anaconda3\lib\runpy.py”, line 193, in _run_module_as_main
“main”, mod_spec)
File “c:\users\sanch\anaconda3\lib\runpy.py”, line 85, in run_code
exec(code, run_globals)
File "C:\Users\sanch\Anaconda3\Scripts\scrapy.exe_main.py", line 9, in
File “c:\users\sanch\anaconda3\lib\site-packages\scrapy\cmdline.py”, line 145, in execute
_run_print_help(parser, _run_command, cmd, args, opts)
File “c:\users\sanch\anaconda3\lib\site-packages\scrapy\cmdline.py”, line 99, in _run_print_help
func(*a, **kw)
File “c:\users\sanch\anaconda3\lib\site-packages\scrapy\cmdline.py”, line 153, in _run_command
cmd.run(args, opts)
File “c:\users\sanch\anaconda3\lib\site-packages\scrapy\commands\crawl.py”, line 57, in run
crawl_defer = self.crawler_process.crawl(spname, **opts.spargs)
File “c:\users\sanch\anaconda3\lib\site-packages\scrapy\crawler.py”, line 176, in crawl
crawler = self.create_crawler(crawler_or_spidercls)
File “c:\users\sanch\anaconda3\lib\site-packages\scrapy\crawler.py”, line 209, in create_crawler
return self._create_crawler(crawler_or_spidercls)
File “c:\users\sanch\anaconda3\lib\site-packages\scrapy\crawler.py”, line 213, in _create_crawler
spidercls = self.spider_loader.load(spidercls)
File “c:\users\sanch\anaconda3\lib\site-packages\scrapy\spiderloader.py”, line 70, in load
raise KeyError(“Spider not found: {}”.format(spider_name))
KeyError: ‘Spider not found: quotes_spider’
PS C:\Users\sanch\Documents\Coding Blocks data science\Scrapy Tutorial\myproject>

Here is the link to the code for quotes_spider.py file

#6

Use “scrapy crawl quotes”

#7

Thanks a lot. It worked this time.
I think I you asked to use ‘quotes’ instead of ‘quotes_spider’ because I initialized name with ‘quotes’

#8

Hey @sanchit123manchanda, yes that was the reason.

Plz mark the doubt as resolved in my doubts section. :blush:

1 Like
#9

hey!
i facing the same problem …
image
still not able to create the file
…can you have a look

#10

image

#11

Hey @Lockhart, refraing from posting on other’s doubts, plz make you separate thread and ask there, i will be replying there only.