paramiko uses tail to get the log output of the server in real time

Basic thinking

At present, there is such a requirement that automation needs to be realized: it is necessary to obtain the temperature of server cpu and gpu and the report of sensor information in real time, and test the influence of high and low temperature environment on the running state of equipment. The basic idea is to use paramiko ssh to the server, start one thread to obtain log output in real time with tail-f command, and start another thread to obtain cpu and gpu temperature regularly with 'cat/sys/class/thermal/thermal_zone0/temp' command.

Code

 def get_report_info_perid(self, cmd, diff_time, thre_time):
  #  Send the command to execute 
  pre_time_stamp = [0] * 4
  self._channel.send(cmd + '\r')
  #  Commands with long echoes may be executed for a long time, and echoes are retrieved in batches through loops 
  time_stamp_arr = []
  index = [0] * 4
  current_line = b''
  line_counter = 0
  line_feed_byte = '\n'.encode(self.encoding)
  while True:
   buffer = self._channel.recv(1)
   if len(buffer) == 0:
    logger.info('end______________')
    break
   current_line += buffer
   if buffer == line_feed_byte:
    line = current_line.decode(self.encoding)
    logger.debug('shell Show: %s'%line)
    if not line.startswith(self.rq):
     line_counter += 1
     current_line = b''
     continue
    col = self.check_type(line)
    time_stamp = int(time.mktime(time.strptime(' '.join([line[:8], line[9:17]]), "%Y%m%d %H:%M:%S")))
    time_stamp_dec = line[18: 21] #  Accurate to milliseconds 
    time_stamp = time_stamp * 1000 + int(time_stamp_dec)
    logger.info('%s:%s' % (senior_name[col], time_stamp))
    self.write_xl(index[col] + 1, col, time_stamp)
    index[col] += 1
    if pre_time_stamp[col] == 0:
     pre_time_stamp[col] = time_stamp
    else:
     if abs((time_stamp - pre_time_stamp[col]) - diff_time[col]) > thre_time[col]:
      logger.error(
       ' The interval between two frames of data is {}ms, The timestamps are :({},{}), Line number: {}'.format(time_stamp - pre_time_stamp[col], time_stamp, pre_time_stamp[col],
                  index[col]))
    pre_time_stamp[col] = time_stamp
    line_counter += 1
    current_line = b''


 def get_temp_info(self, col, max_number):
  index = 0
  cpu_arr, gpu_arr = [], []
  while True:
   cpu_temp, gpu_temp = self.get_cpu_gpu_temp()
   logger.info('cpu_temp:%s, gpu_temp:%s' % (cpu_temp, gpu_temp))
   cpu_arr.append(cpu_temp)
   gpu_arr.append(gpu_temp)
   self.write_xl(index + 1, col, cpu_temp)
   self.write_xl(index + 1, col + 1, cpu_temp)
   time.sleep(60)
   index += 1
   if max_number == index:
    break
  return cpu_arr, gpu_arr

Encounter problems

1. Question 1

1 The starting cmd command is tail-f log. txt grep-aE "ab"

As a result, there is a problem, and after a few minutes of code running, no data can be obtained

At first, 1 thought it was a problem of paramiko, which would automatically shut down client after a fixed time, but after debugging, it was found that it was blocked at _ channel. recv, and 1 could not receive the data from the server directly.

After Baidu, it is found that due to the buffer mechanism of linux, the output delay will occur when tail-f is combined with the pipeline

Buffering is an effective method to improve the efficiency of IO, which accumulates frequent read and write requests to a certain degree and then interacts with IO devices once.

There are three types of IO buffers, no buffering, line buffering, and full buffering.

No buffering means no buffering mechanism is used. Byte-oriented device? (stderr) Line buffering, buffering, until a line break is encountered. 1 is generally used for terminal equipment. Full buffer, buffer until buffer is full. 1 is generally used for block devices.

Execute the tail command in the terminal window, which is oriented to the terminal device and will use line buffering, so every line written in the log will be output immediately.

When pipes are used, full buffering will be used, so that 1 will not be output until the number of bytes written in the log fills buffer.

Solution:

Redirect the standard output of tail to the standard error, and give the standard error to the pipeline as well.

Because stderr is unbuffered.

E.g. tail-f > & 2 | grep

Or just remove the pipe

2. Question 2

According to the conclusion of question 1, I removed the pipeline in the command, directly used the command tail-f log. txt, and put the filtering into the function check_type, and found that the situation that no data was obtained after running for several minutes was not solved. So continue to locate. Finally, after 1 setback, I found that there was something wrong with the tail-f command

tail -f

follow=descriptor, tracing based on file descriptor. Tracing stops when file is renamed or deleted

tail -F

follow=name--retry, tracing by filename and keep retrying, that is, after the file is deleted or renamed, tracing continues if the same filename is created again

The log. txt file has its file descriptor modified during program running so that tail-f does not continue tracing. Problem resolution after modification to tail-F