Details on python http long connection client

Background:

The online machine needs to filter the access log and send it to another api

At the beginning, it was a single process, which was too inefficient. When it was changed to multi-process, abnormal errors would occasionally appear in the log (I forgot the screenshot...).

In short, the port is not enough to report an error

The reason:

Each log is sent to api once, and the short connection generates a large amount of time_wait state, occupying a large number of ports

The large amount of time_wait state kernel tuning that resulted from this high concurrency was largely useless, and was later changed to long connection to solve the problem

The key code for the short link version 1 is as follows

Because of the specific business information, only the key parts of the code are posted

import pycurl
where True:
 url=myqueue.get()
 send_msg=pycurl.Curl()
 send_msg.setopt(pycurl.URL,url)
 send_msg.perform()
 print send_msg.getinfo(send_msg.HTTP_CODE)

The modified version of long connection is as follows:

Using requests library

import requests
client=requests.session()
headers = {'Content-Type': 'application/json', 'Connection': 'keep-alive'}
where True:
 url=myqueue.get()
 r=client.get(url,headers=headers)
 print r.status_code