Solve the problem of repeated printing of pit log encountered by python logging

If logging module in python encounters multi-thread or multi-process or customizes logging in web framework (one request is one independent thread), it is very easy to print logs repeatedly and cause memory crash, so:

The solution is as follows:

Class for overriding log methods:

 class Log():
 import logging
 def __init__(self):
 self.logger = logging.getLogger(__name__)
 #  The following 3 Behavior empties the last file 
 #  Object that empties the current file logging  Because logging Will contain all the files logging
 logging.Logger.manager.loggerDict.pop(__name__)
 #  Object of the current file handlers  Empty  
 self.logger.handlers = []
 #  Then remove the current file again logging Configure 
 self.logger.removeHandler(self.logger.handlers)
 #  To judge here, if logger.handlers If the list is empty, add it; Otherwise, write the log directly 
 if not self.logger.handlers:
 # loggger  File configuration path 
 self.handler = logging.FileHandler(os.getcwd() + '/logger/%s_log/%s_score.log' % (str(dt.date.today()), str(dt.date.today())))
 # logger  Configuration level 
 self.logger.setLevel(logging.DEBUG)
 # logger  Output format 
 formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(name)s - %(message)s')
 #  Add output format to enter handler
 self.handler.setFormatter(formatter)
 #  Add file settings such as handler
 self.logger.addHandler(self.handler)
 
 #  The following are override methods   And clear after each record logger
 def info(self,message=None):
 self.__init__()
 self.logger.info(message)
 self.logger.removeHandler(self.logger.handlers)
 
 def debug(self,message=None):
 self.__init__()
 self.logger.debug(message)
 self.logger.removeHandler(self.logger.handlers)
 
 def warning(self,message=None):
 self.__init__()
 self.logger.warning(message)
 self.logger.removeHandler(self.logger.handlers)
 
 def error(self,message=None):
 self.__init__()
 self.logger.error(message)
 self.logger.removeHandler(self.logger.handlers)
 
 def critical(self, message=None):
 self.__init__()
 self.logger.critical(message)
 self.logger.removeHandler(self.logger.handlers)

Pro-test is effective!

In addition, the module pays special attention to the fact that, for example, when web requests, it is called at the interface and then the boot pass parameter must not be a global variable

Supplement: python, multiple files share logger, and the solution to the problem of repeated printing

Problem background & Phenomenon

In a recent project, you need to use the logging library of python to print the log to a file, and then put the python script into crontab for execution. So I wrote a simple package of logger.

As follows:

#!/usr/bin/python
# -*- coding:utf-8 -*- 
import logging
import time
import os 
class Log(object):
 '''
 Encapsulated logging
 ''' 
 def __init__(self, logger=None, log_cate='search'):
  '''
    Specify the file path to save the log, the log level, and the call file 
    Save the log to the specified file 
  ''' 
  #  Create 1 A logger
  self.logger = logging.getLogger(logger)
  self.logger.setLevel(logging.DEBUG)
  #  Create 1 A handler For writing to log files 
  self.log_time = time.strftime("%Y_%m_%d")
  file_dir = os.getcwd() + '/../log'
  if not os.path.exists(file_dir):
   os.mkdir(file_dir)
  self.log_path = file_dir
  self.log_name = self.log_path + "/" + log_cate + "." + self.log_time + '.log'
  # print(self.log_name)
 
  fh = logging.FileHandler(self.log_name, 'a') #  Append mode   This is python2 Adj. 
  # fh = logging.FileHandler(self.log_name, 'a', encoding='utf-8') #  This is python3 Adj. 
  fh.setLevel(logging.INFO)
 
  #  Re-create 1 A handler For output to the console 
  ch = logging.StreamHandler()
  ch.setLevel(logging.INFO)
 
  #  Definition handler Output format of 
  formatter = logging.Formatter(
   '[%(asctime)s] %(filename)s->%(funcName)s line:%(lineno)d [%(levelname)s]%(message)s')
  fh.setFormatter(formatter)
  ch.setFormatter(formatter)
 
  #  To logger Add handler
  self.logger.addHandler(fh)
  self.logger.addHandler(ch)
 
  #  Add the following 1 Sentence to remove the handle after logging 
  # self.logger.removeHandler(ch)
  # self.logger.removeHandler(fh)
  #  Close an open file 
  fh.close()
  ch.close()
 
 def getlog(self):
  return self.logger

The purpose is to make all the places where logger is used, only import is the encapsulation library, and then call it directly. For example, calling logger

a.py

#!/usr/bin/python
# -*- coding:utf-8 -*-
 
from common.log import Log
log = Log().getlog()
log.info("I am a.py")

b.py

#!/usr/bin/python
# -*- coding:utf-8 -*-
 
from common.log import Log
log = Log().getlog()
log.info("I am b.py")

c.py

#!/usr/bin/python
# -*- coding:utf-8 -*-
 
import a
import b
from common.log import Log 
log = Log().getlog()
log.info("I am c.py")

The result of executing c. py at this point is as follows:

➜ search git:(master) ✗ python c.py

[2019-01-14 15:58:35,807] a.py- > < module > line:6 [INFO]I am a.py

[2019-01-14 15:58:35,808] b.py- > < module > line:6 [INFO]I am b.py

[2019-01-14 15:58:35,808] b.py- > < module > line:6 [INFO]I am b.py

[2019-01-14 15:58:35,809] c.py- > < module > line:8 [INFO]I am c.py

[2019-01-14 15:58:35,809] c.py- > < module > line:8 [INFO]I am c.py

[2019-01-14 15:58:35,809] c.py- > < module > line:8 [INFO]I am c.py

It can be seen that logger of a. py, b. py and c. py are shared, resulting in repeated printing.

Cause analysis of the problem

From the phenomenon, it can be concluded that log systems between different files influence each other. In a. py, b. py, c. py, our calling mode is log = Log (). getlog (), that is, self. logger = logging. getLogger (logger), and logger parameters are not passed, so the resulting self. logger is RootLogger.

RootLogger is the globally unique ancestor of all Logger objects within an python program. Therefore, our setting of RootLogger will naturally affect all log output. In short, it is the setting of log in the first opened file, and the later opened file will be affected, and the inheritance relationship of logger will be passed through once. In this example, b. py is followed by import after a. py, so b. py executes its own logger once, and then executes RootLogger opened in a. py, and so on..........

Problem solving method

Instead of the default RootLogger, give each Logger a name.

a.py

from common.log import Log
log = Log(__name__).getlog()
log.info("I am a.py")

b.py

from common.log import Log
log = Log(__name__).getlog()
log.info("I am b.py")

c.py

import b
import a 
from common.log import Log 
log = Log(__name__).getlog()
log.info("I am c.py")

The latest implementation results of c. py:

➜ search git:(master) ✗ python c.py

[2019-01-14 16:24:12,008] b.py- > < module > line:6 [INFO]I am b.py

[2019-01-14 16:24:12,009] a.py- > < module > line:6 [INFO]I am a.py

[2019-01-14 16:24:12,009] c.py- > < module > line:10 [INFO]I am c.py

There is no repetition, which is in line with expectations. The problem was solved.