A simple crawler code for python3

  • 2020-04-02 13:41:54
  • OfStack

I have to say that python is pretty easy to get started with. Looked up on the Internet, is mostly python2 posts, so casually wrote a python3. The code is very simple don't explain, just paste the code.


#test rdp
import urllib.request
import re<br>
# Login account information 
data={}
data['fromUrl']=''
data['fromUrlTemp']=''
data['loginId']='12345'
data['password']='12345'
user_agent='Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
# The login address 
#url='http://192.168.1.111:8080/loginCheck'
postdata = urllib.parse.urlencode(data) 
postdata = postdata.encode('utf-8')
headers = { 'User-Agent' : user_agent }
# The login  
res = urllib.request.urlopen(url,postdata)
# Get the page html<br>strResult=(res.read().decode('utf-8'))
# Take all of them with a regular expression A The label 
p = re.compile(r'<a href="(.*?)".*?>(.*?)</a>')
for m in p.finditer(strResult):
    print (m.group(1))#group(1) is href What's in there, group(2) is a The text in the label 

I took a look at the processing of cookies, exceptions and so on, but I didn't spend any time to deal with them. After all, I just wanted to learn python by writing crawlers.


Related articles: