Detail Python ES1en. Popen subprocess pipeline block

Problem generation description

Use the child process to process a large log file, and analyze and query the file. You need to wait for the output result executed by the child process for the next step.

Problem code

#  Enable child processes to execute externally shell The command 
def __subprocess(self,cmd):
 try:
 #  Performing external shell Command,   Output result Output pipe 
 p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) 
 p.wait()

 #  Read from standard output shell The output of the command 
 #rt = p.stdout.read().decode()

 #  Break up the data with newline characters and drop newline characters into the list 
 rt_list = rt.strip().split('\n')

 except Exception as e:
 if(DEBUG):
  print(traceback.format_exc())

 return rt_list

Problem analysis

When buffer is full, it writes to the child's stdout and stdout, which are piped to the parent process. When the pipe is full, the child process stops writing, becomes stuck, and the output from taking the pipe out in time does not block

However, what I take here is temporary file receiving sub-process output. Since temporary file is built on disk, there is no limitation of size, and after the file is close, the corresponding space on disk will also be freed.

Improved code

import tempfile
#  Enable child processes to execute externally shell The command 
def __subprocess(self,cmd):
 try:
 #  get 1 A temporary file object,   call close The file is deleted from disk 
 out_temp = tempfile.TemporaryFile(mode='w+')
 #  Gets the file number of a temporary file 
 fileno = out_temp.fileno()

 #  Performing external shell Command,   The output is stored in a temporary file 
 p = subprocess.Popen(cmd, shell=True, stdout=fileno, stderr=fileno) 
 p.wait()

 #  Read from a temporary file shell The output of the command 
 out_temp.seek(0)
 rt = out_temp.read()

 #  Break up the data with newline characters and drop newline characters into the list 
 rt_list = rt.strip().split('\n')

 except Exception as e:
 if(DEBUG):
  print(traceback.format_exc())

 finally:
 if out_temp:
  out_temp.close()

 return rt_list