Web scraping related

#1

android_data=urlopen(android_url)
It says CERTIFICATE VERIFICATION FAILED.

#2

Requesting you to attach the code before the problematic line.
Coming to the mentioned error, There are two possible ways to deal with this:

  • This error is sometimes caused by a specific python version. One quick fix is to create a new conda or pip environment with a different Python version and try running the code again. e.g.
    conda create -n env python=3.7

  • Or you can bypass the certificate verification step altogether (not recommended). Add this piece of code before making the https request:
    import os, ssl
    if ( not os.environ.get(‘PYTHONHTTPSVERIFY’, ‘’) and
    getattr(ssl, ‘_create_unverified_context’, None)):
    ssl._create_default_https_context = ssl._create_unverified_context

Reference:


Let me know if this helps.

#3

Thank you so much for replying. It’s working on Python 2.7 but not on 3. Don’t really know what might need a fix…

#4

Have you tried adding this snippet at the top of your code?

import os, ssl
if (not os.environ.get('PYTHONHTTPSVERIFY', '') and getattr(ssl, '_create_unverified_context', None)):
    ssl._create_default_https_context = ssl._create_unverified_context

This basically bypasses the certificate validation step.

#5

now its not showing CERT Verific error. But when I execute
android_html=android_data.read()
print(android_html)

it shows “b” and not the html code

#6

Requesting you to verify if the URL is correct:
“https://en.wikipedia.org/wiki/Android_version_history”
Also, it would be helpful if you could share the complete code. Just paste the code at cb.lk/ide, save it, and share the generated URL.

#7

I hope I’ve cleared your doubt. I ask you to please rate your experience here
Your feedback is very important. It helps us improve our platform and hence provide you
the learning experience you deserve.

On the off chance, you still have some questions or not find the answers satisfactory, you may reopen
the doubt.