The python socket timeout setting errno 10054

Python socket.error: [Errno 10054] the remote host forced an existing connection to close. Solution to the problem:

The other day I used python to read web pages. Because of the heavy use of the urlopen operation on a site, it is considered an attack by that site. Sometimes downloads are no longer allowed. After causing urlopen(), requeste.read () remains stuck. Finally, errno 10054 is thrown.

The error is connection reset by peer. The reason may be socket timeout time is too long; It could also be that after request = urllib.request-urlopen (url), there is no request-close () operation. It could also be that there is no sleep for a few seconds, leading the site to conclude that this behavior is an attack.

The specific solution is as follows:

01.import socket 
02.import time 
03.timeout = 20  
04.socket.setdefaulttimeout(timeout)# Here for the whole socket The layer sets the timeout time. If used again in subsequent files socket , no need to set   
05.sleep_download_time = 10 
06.time.sleep(sleep_download_time) # Here the time is set by itself   
07.request = urllib.request.urlopen(url)# This is where the content is going to be read url  
08.content = request.read()# Read, usually an exception will be reported here   
09.request.close()# Remember to close it  

Because the read() operation after urlopen is actually calling some function of the socket layer. So set the socket default timeout time, you can let the network break itself. You don't have to wait forever at read().

Of course, you can also write several try outside,except, for example:

try: 
  time.sleep(self.sleep_download_time) 
  request = urllib.request.urlopen(url) 
  content = request.read() 
  request.close() 
   
except UnicodeDecodeError as e: 
     
  print('-----UnicodeDecodeError url:',url) 
   
except urllib.error.URLError as e: 
  print("-----urlError url:",url) 
 
except socket.timeout as e: 
  print("-----socket timout:",url) 

Generally speaking, there is no problem. I tested thousands of web downloads before I said this. But if I downloaded thousands of them, and I tested them, ms would still jump out of this exception. It could be time. Sleep () is too short, or it could be a sudden network outage. I tested it with urllib.request-retrieve () and found that there were always failures as I kept downloading data.

The easy way to do this is to first refer to my article: (link: #). Do a checkpoint first. Then, while True the above bit of code that will run an exception. See the pseudocode below:

def Download_auto(downloadlist,fun,sleep_time=15): 
  while True:     
    try: #  A layer of outsourcing try  
      value = fun(downloadlist,sleep_time) #  Here, fun Is your download function, I pass in when the function pointer.  
      #  Only normal execution can exit.   
      if value == Util.SUCCESS: 
        break 
    except : #  If that happens 10054 or IOError or XXXError 
      sleep_time += 5 # More sleep 5 Second, re-execute the above download. Because of the checkpoint, the above program continues execution from where the exception was thrown. Prevents program interruption due to unstable network connection.  
      print('enlarge sleep time:',sleep_time) 

However, for the corresponding web page, and to do another processing:

#  Print download information   
def reporthook(blocks_read, block_size, total_size): 
  if not blocks_read: 
    print ('Connection opened') 
  if total_size < 0: 
    print ('Read %d blocks' % blocks_read) 
  else: 
    #  If you can't find it, the page doesn't exist, maybe totalsize is 0 Cannot calculate the percentage   
    print('downloading:%d MB, totalsize:%d MB' % (blocks_read*block_size/1048576.0,total_size/1048576.0)) 
    
 
def Download(path,url): 
#url = 'http://downloads.sourceforge.net/sourceforge/alliancep2p/Alliance-v1.0.6.jar' 
  #filename = url.rsplit("/")[-1]  
  try: 
  # python The built-in download function   
    urllib.request.urlretrieve(url, path, reporthook) 
  except IOError as e: #  If you can't find it, it's going to cause it IOError .  
    print("download ",url,"/nerror:",e) 
  print("Done:%s/nCopy to:%s" %(url,path)) 

If you still have a problem... Please note other solutions in the comments.