Python USES the proxy server method for data collection based on scrapy

This article illustrates the proxy server method used by Python to collect data based on scrapy. Share with you for your reference. The details are as follows:

# To authenticate the proxy, 
#you must set the Proxy-Authorization header. 
#You *cannot* use the form http://user:pass@proxy:port 
#in request.meta['proxy']
import base64
proxy_ip_port = "123.456.789.10:8888"
proxy_user_pass = "awesome:dude"
request = Request(url, callback=self.parse)
# Set the location of the proxy
request.meta['proxy'] = "http://%s" % proxy_ip_port
# setup basic authentication for the proxy
encoded_user_pass=base64.encodestring(proxy_user_pass)
request.headers['Proxy-Authorization'] = 'Basic ' + encoded_user_pass

I hope this article has been helpful to your python programming.