USER_AGENT (permission to access on pepperfry server)

#1

If I am doing this project by using BeautifulSoup (bs4), then how I will get permission to get data ?

#2

hey @mayanksingh4597 ,
Search for the file settings.py in your scrapy project.
In that file, add these two lines of code:

USER_AGENT = 'Mozilla/5.0'
ROBOTSTXT_OBEY = False

This will resolve the issue.

#3

You are giving the way to grant access if I am doing it with scrapy.
But I am asking for beautiful soup not scrapy.

#4

ohh sorry , thats my mistake.
With beautifulsoup , request the data from website using requests module and pass a parameter headers with it , like this.

headers = {'User-Agent': 'Mozilla/5.0'}

url = https://www.somthing.com

r = requests.get(url, headers=headers)

I hope this helps you :slightly_smiling_face:.