Details of the exception retry solution in Python

preface

When we are doing data capture, we often encounter the program saving caused by network problems. We only recorded the wrong content before, and processed the wrong content later.

The original process:

def crawl_page(url):
 pass
 
def log_error(url):
 pass
 
url = ""
try:
 crawl_page(url)
except:
 log_error(url)

The improved process:

attempts = 0
success = False
while attempts < 3 and not success:
 try:
  crawl_page(url)
  success = True
 except:
  attempts += 1
  if attempts == 3:
   break

A recently discovered new solution: retrying

retrying is a retry package of Python that can be used to automatically retry segments that may fail to run. retrying Provides a decorator function retry , the decorated function will be reexecuted if it fails to run. By default, the function will be retried as long as 1 is reported.

import random
from retrying import retry
 
@retry
def do_something_unreliable():
 if random.randint(0, 10) > 1:
  raise IOError("Broken sauce, everything is hosed!!!111one")
 else:
  return "Awesome sauce!"
 
print do_something_unreliable()

If we run have_a_try Delta function, so it goes all the way to random.randint Return 5 and it will finish execution, otherwise 1 will resume execution.

retry can also accept 1 parameter, which is optional from the initialization function of the Retrying class in the source code:

stop_max_attempt_number : is used to set the maximum number of attempts, beyond which the retry will stop stop_max_delay : for example, if set to 10000, the function will not be executed as long as it is longer than 10 seconds from the time when the decorated function starts to execute, to the time when the function successfully runs or aborts due to failure wait_fixed : set twice retrying Stay time between wait_random_min和wait_random_max : produced twice in a random fashion retrying Stay time between retry0 : produced twice in an exponential form retrying And the value generated is 2^previous_attempt_number * wait_exponential_multiplier . previous_attempt_number It's already retry The number of times, if this value is more than 1 wait_exponential_max , then the rest value between the two retrying is wait_exponential_max . This design caters exponential backoff Algorithm, can reduce the blocking situation. We can specify which exceptions we want to go to retry, which we'll use retry_on_exception Pass in 1 function object:

def retry_if_io_error(exception):
 return isinstance(exception, IOError)
 
@retry(retry_on_exception=retry_if_io_error)
def read_a_file():
 with open("file", "r") as f:
  return f.read()

In the implementation read_a_file During the function, if an exception is reported, the exception will take the form parameter exception The incoming retry_if_io_error In the function if exception is IOError So let's go ahead retry If not, stop running and throw an exception.

We can also specify which results to go with retry I'm going to use this retry_on_result Pass in 1 function object:

def retry_if_result_none(result):
 return result is None
 
@retry(retry_on_result=retry_if_result_none)
def get_result():
 return None

In the implementation get_result On success, the return value of the function is passed through the parameter result In the form of retry_if_result_none In the function, if the return value is random.randint0 So let's go ahead retry , otherwise it ends and returns the value of the function.

conclusion