Implement a simple project monitoring using Python

An interface system made in a company, which is mainly to connect with the system interface of a third party, so the system will interact with the projects of many other companies. Following a very painful problem, so many company interface, the stability of different company interface is very different, when the traffic is large, some not very line interface on a variety of errors.

This interface system has just been developed, and it is on the edge of the whole system, unlike other projects, where there is a log library, and there is a message alert, and if something goes wrong, a lot of times it is the user feedback, so my idea is to pick up python and write a monitor for this project. If there are a lot of errors in the process of invoking a third-party interface, that means there is something wrong with the interface and you can take action more quickly.

The project also has a log library. All the info and error logs are scanned and stored every one minute. The log library is mysql.

              Level log level               Message log content               File_name Java code file               Log_time logs time

Log library, do not need to scan their own online environment log analysis, directly from the log library. Since the log library is on the line every 1 minute, I will go to the log library every 2 minutes to scan, if there is a certain number of error log will alarm, if only one or two errors can be ignored, that is, a large number of error log in a short time, you can conclude that there is a problem with the system. The alarm is sent via email, so you need to do the following:
1. Operate MySql.
Send an email.
3. Schedule tasks.
4. Log.
5. Run the script.

Once you've identified the above few things, you can get started.
Operational database

Use MySQLdb this driver, direct operation database, mainly is the query operation.
Get a connection to the database:
 

def get_con():
 host = "127.0.0.1"
 port = 3306
 logsdb = "logsdb"
 user = "root"
 password = "never tell you"
 con = MySQLdb.connect(host=host, user=user, passwd=password, db=logsdb, port=port, charset="utf8")
 return con

Get the data from the log library, get the data 2 minutes before the current time, first, calculate the time according to the current time. Before, there was a problem with the calculation, which has been modified.
 

def calculate_time():
 
 now = time.mktime(datetime.now().timetuple())-60*2
 result = time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(now))
 return result

The log library is then queried for data based on time and log levels
 

def get_data():
 select_time = calculate_time()
 logger.info("select time:"+select_time)
 sql = "select file_name,message from logsdb.app_logs_record " 
   "where log_time >"+"'"+select_time+"'" 
   "and level="+"'ERROR'" 
   "order by log_time desc"
 conn = get_con()
 
 cursor = conn.cursor()
 cursor.execute(sql)
 results = cursor.fetchall()
 
 cursor.close()
 conn.close()
 
 return results

Send E-mail

Using python to send mail is relatively simple, using the standard library smtplib can be
You can use any other mailbox or enterprise mailbox, but the host and port should be set correctly.
 

def send_email(content):
 
sender = "sender_monitor@163.com"
receiver = ["rec01@163.com", "rec02@163.com"]
host = 'smtp.163.com'
port = 465
msg = MIMEText(content)
msg['From'] = "sender_monitor@163.com"
msg['To'] = "rec01@163.com,rec02@163.com"
msg['Subject'] = "system error warning"
 
try:
smtp = smtplib.SMTP_SSL(host, port)
smtp.login(sender, '123456')
smtp.sendmail(sender, receiver, msg.as_string())
logger.info("send email success")
except Exception, e:
logger.error(e)

Timing task

Use a separate thread, scan every 2 minutes, and send an email if the error-level number of logs exceeds 5.
 

def task():
while True:
logger.info("monitor running")
 
results = get_data()
if results is not None and len(results) > 5:
content = "recharge error : "
logger.info("a lot of error,so send mail")
for r in results:
content += r[1]+'n'
send_email(content)
sleep(2*60)

The log

Configure the log log.py for this little script so that the log can be output to files and the console.

# coding=utf-8
import logging
 
logger = logging.getLogger('mylogger')
logger.setLevel(logging.DEBUG)
 
fh = logging.FileHandler('monitor.log')
fh.setLevel(logging.INFO)
 
ch = logging.StreamHandler()
ch.setLevel(logging.INFO)
 
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fh.setFormatter(formatter)
ch.setFormatter(formatter)
 
logger.addHandler(fh)
logger.addHandler(ch)

So, in the end, this monitoring applet is just like app_monitor.py

# coding=utf-8
import threading
import MySQLdb
from datetime import datetime
import time
import smtplib
from email.mime.text import MIMEText
from log import logger
 
 
def get_con():
 host = "127.0.0.1"
 port = 3306
 logsdb = "logsdb"
 user = "root"
 password = "never tell you"
 con = MySQLdb.connect(host=host, user=user, passwd=password, db=logsdb, port=port, charset="utf8")
 return con
 
 
def calculate_time():
 
 now = time.mktime(datetime.now().timetuple())-60*2
 result = time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(now))
 return result
 
 
def get_data():
 select_time = calculate_time()
 logger.info("select time:"+select_time)
 sql = "select file_name,message from logsdb.app_logs_record " 
   "where log_time >"+"'"+select_time+"'" 
   "and level="+"'ERROR'" 
   "order by log_time desc"
 conn = get_con()
 
 cursor = conn.cursor()
 cursor.execute(sql)
 results = cursor.fetchall()
 
 cursor.close()
 conn.close()
 
 return results
 
 
def send_email(content):
 
 sender = "sender_monitor@163.com"
 receiver = ["rec01@163.com", "rec02@163.com"]
 host = 'smtp.163.com'
 port = 465
 msg = MIMEText(content)
 msg['From'] = "sender_monitor@163.com"
 msg['To'] = "rec01@163.com,rec02@163.com"
 msg['Subject'] = "system error warning"
 
 try:
  smtp = smtplib.SMTP_SSL(host, port)
  smtp.login(sender, '123456')
  smtp.sendmail(sender, receiver, msg.as_string())
  logger.info("send email success")
 except Exception, e:
  logger.error(e)
 
 
def task():
 while True:
  logger.info("monitor running")
  results = get_data()
  if results is not None and len(results) > 5:
   content = "recharge error : "
   logger.info("a lot of error,so send mail")
   for r in results:
    content += r[1]+'n'
   send_email(content)
  time.sleep(2*60)
 
 
def run_monitor():
 monitor = threading.Thread(target=task)
 monitor.start()
 
 
if __name__ == "__main__":
 run_monitor()

Run the script

The script runs on the server, using the supervisor for management.
Install supervisor on the server (centos6) and then add the following configuration in /etc/supervisor. Conf:

[program:app-monitor]

command = python /root/monitor/app_monitor.py

directory = /root/monitor

user = root

Then run supervisor on the terminal by running supervisor ord.
Run the supervisory CTL in the terminal, go into the shell, and run status to see how the script is running.
conclusion

This small monitoring idea is very clear, still can continue to modify, for example: monitoring specific interface, send SMS notification and so on.
Because there is a log library, it is less trouble to scan the log in the online formal environment, so, if there is no log library, it is necessary to scan the online environment, in the formal online environment must be careful