Python urlopen of function sample sharing
- 2020-04-02 13:47:15
- OfStack
Well, without further ado, let's look at some examples
One, open a web page to get all the content
from urllib import urlopen
doc = urlopen("http://www.baidu.com").read()
print doc
Get the Http header
from urllib import urlopen
doc = urlopen("http://www.baidu.com")
print doc.info()
print doc.info().getheader('Content-Type')
Use agents
1. Look at the environment variables
print ""n".join(["%s=%s" % (k, v) for k, v in os.environ.items()])
print os.getenv("http_proxy")
2. Set environment variables
import os
os.putenv("http_proxy", "http://proxyaddr:<port>")
Use an agent
# Use http://www.someproxy.com:3128 for http proxying
proxies = {'http': 'http://www.someproxy.com:3128'}
filehandle = urllib.urlopen(some_url, proxies=proxies)
# Don't use any proxies
filehandle = urllib.urlopen(some_url, proxies={})
# Use proxies from environment - both versions are equivalent
filehandle = urllib.urlopen(some_url, proxies=None)
filehandle = urllib.urlopen(some_url)