Implementation example of obtaining camera and controlling face in real time with Python

Implementation process

Get the video stream from the camera and convert it into a frame-by-frame image, then pass the image information to the opencv tool library for processing, and return a gray image (just like you use a local still picture 1)

After the program starts, according to the listener information, use an while loop to load the video image continuously, and then return to the opencv tool to present the image information.

Create a keyboard event monitor, press the "d" key, and start face matching and mask loading (this process is dynamic and you can move at any time).

Face matching uses the face detection algorithm in Dlib to see if there is a face. If so, it will create an end position for each face, where glasses and cigarettes will move to end.

Then we need to zoom and rotate our glasses to fit everyone's face. We will use the point set returned from Dlib's 68-point model to find the center of the eyes and mouth and rotate the space between them.

After we get the final position of glasses and cigarettes in real time, glasses and cigarettes enter from the top of the screen and start matching your glasses and mouth.

If there is no face, the program will return your video information directly, and there will be no mask moving effect.

The default 1 cycle is 4 seconds. Then you can check again with the "d" key.

The program exits using the "q" key.

Here I abstracted this function into a mask loading service, please follow my code 1 to see what happened.

1. Import the corresponding toolkit

from time import sleep

import cv2
import numpy as np
from PIL import Image
from imutils import face_utils, resize

try:
  from dlib import get_frontal_face_detector, shape_predictor
except ImportError:
  raise

Create the mask loading service class DynamicStreamMaskService and its corresponding initialization properties:

class DynamicStreamMaskService(object):
  """
   Dynamic paste mask service 
  """

  def __init__(self, saved=False):
    self.saved = saved #  Do you want to save pictures 
    self.listener = True #  Startup parameters 
    self.video_capture = cv2.VideoCapture(0) #  Call the local camera 
    self.doing = False #  Do you have a facial mask 
    self.speed = 0.1 #  Moving speed of mask 
    self.detector = get_frontal_face_detector() #  Facial recognizer 
    self.predictor = shape_predictor("shape_predictor_68_face_landmarks.dat") #  Facial analyzer 
    self.fps = 4 #  Base time of mask existence time 
    self.animation_time = 0 #  Initial value of animation period 
    self.duration = self.fps * 4 #  Maximum animation period 
    self.fixed_time = 4 #  After drawing, stay time 
    self.max_width = 500 #  Image size 
    self.deal, self.text, self.cigarette = None, None, None #  Mask object 

According to the above introduction, we first realize the function of reading video stream and converting pictures:

def read_data(self):
  """
   Get the video stream from the camera and convert it to 1 Frame 1 Frame image 
  :return:  Return 1 Frame 1 Image information of frames 
  """
  _, data = self.video_capture.read()
  return data

Next, we realize the face location function and the location of glasses and cigarettes:

def get_glasses_info(self, face_shape, face_width):
  """
   Obtain glasses information of the current face 
  :param face_shape:
  :param face_width:
  :return:
  """
  left_eye = face_shape[36:42]
  right_eye = face_shape[42:48]

  left_eye_center = left_eye.mean(axis=0).astype("int")
  right_eye_center = right_eye.mean(axis=0).astype("int")

  y = left_eye_center[1] - right_eye_center[1]
  x = left_eye_center[0] - right_eye_center[0]
  eye_angle = np.rad2deg(np.arctan2(y, x))

  deal = self.deal.resize(
    (face_width, int(face_width * self.deal.size[1] / self.deal.size[0])),
    resample=Image.LANCZOS)

  deal = deal.rotate(eye_angle, expand=True)
  deal = deal.transpose(Image.FLIP_TOP_BOTTOM)

  left_eye_x = left_eye[0, 0] - face_width // 4
  left_eye_y = left_eye[0, 1] - face_width // 6

  return {"image": deal, "pos": (left_eye_x, left_eye_y)}

def get_cigarette_info(self, face_shape, face_width):
  """
   Get the cigarette information of the current face 
  :param face_shape:
  :param face_width:
  :return:
  """
  mouth = face_shape[49:68]
  mouth_center = mouth.mean(axis=0).astype("int")
  cigarette = self.cigarette.resize(
    (face_width, int(face_width * self.cigarette.size[1] / self.cigarette.size[0])),
    resample=Image.LANCZOS)
  x = mouth[0, 0] - face_width + int(16 * face_width / self.cigarette.size[0])
  y = mouth_center[1]
  return {"image": cigarette, "pos": (x, y)}

def orientation(self, rects, img_gray):
  """
   Face location 
  :return:
  """
  faces = []
  for rect in rects:
    face = {}
    face_shades_width = rect.right() - rect.left()
    predictor_shape = self.predictor(img_gray, rect)
    face_shape = face_utils.shape_to_np(predictor_shape)
    face['cigarette'] = self.get_cigarette_info(face_shape, face_shades_width)
    face['glasses'] = self.get_glasses_info(face_shape, face_shades_width)

    faces.append(face)

  return faces

Just now we mentioned the keyboard listening event, and here we implement this function under 1:

def listener_keys(self):
  """
   Set keyboard listening events 
  :return:
  """
  key = cv2.waitKey(1) & 0xFF
  if key == ord("q"):
    self.listener = False
    self.console(" Program exit ")
    sleep(1)
    self.exit()

  if key == ord("d"):
    self.doing = not self.doing

Next, let's implement the function of loading mask information:

def init_mask(self):
  """
   Load mask 
  :return:
  """
  self.console(" Load mask ...")
  self.deal, self.text, self.cigarette = (
    Image.open(x) for x in ["images/deals.png", "images/text.png", "images/cigarette.png"]
  )

All the above basic functions have been realized, so it's time for us to realize the drawing function. The principle of this function is the same as the one I wrote before, which uses AI face recognition technology to realize Tik Tok special effects. I won't repeat it here. You can go to github or Python Chinese community WeChat official account to check it.

def drawing(self, draw_img, faces):
  """
   Draw a picture 
  :param draw_img:
  :param faces:
  :return:
  """
  for face in faces:
    if self.animation_time < self.duration - self.fixed_time:
      current_x = int(face["glasses"]["pos"][0])
      current_y = int(face["glasses"]["pos"][1] * self.animation_time / (self.duration - self.fixed_time))
      draw_img.paste(face["glasses"]["image"], (current_x, current_y), face["glasses"]["image"])

      cigarette_x = int(face["cigarette"]["pos"][0])
      cigarette_y = int(face["cigarette"]["pos"][1] * self.animation_time / (self.duration - self.fixed_time))
      draw_img.paste(face["cigarette"]["image"], (cigarette_x, cigarette_y),
              face["cigarette"]["image"])
    else:
      draw_img.paste(face["glasses"]["image"], face["glasses"]["pos"], face["glasses"]["image"])
      draw_img.paste(face["cigarette"]["image"], face["cigarette"]["pos"], face["cigarette"]["image"])
      draw_img.paste(self.text, (75, draw_img.height // 2 + 128), self.text)

Since it is a service class, there should be start and exit functions. Finally, let's write 1.

Brief introduction of 1 under this start () function, after starting according to the initialization of monitoring information, constantly monitoring video streams, and stream information through opencv converted to image display.
And call the key monitoring function to continuously monitor whether you press the "d" key to load the mask. If the monitoring is successful, the image face detection is carried out and the mask is moved.
At the end of one cycle, the mask will move according to your face movement. Finally, the effect of the picture at the top of the article is presented.

def start(self):
  """
   Start program 
  :return:
  """
  self.console(" Program started successfully .")
  self.init_mask()
  while self.listener:
    frame = self.read_data()
    frame = resize(frame, width=self.max_width)
    img_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    rects = self.detector(img_gray, 0)
    faces = self.orientation(rects, img_gray)
    draw_img = Image.fromarray(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))
    if self.doing:
      self.drawing(draw_img, faces)
      self.animation_time += self.speed
      self.save_data(draw_img)
      if self.animation_time > self.duration:
        self.doing = False
        self.animation_time = 0
      else:
        frame = cv2.cvtColor(np.asarray(draw_img), cv2.COLOR_RGB2BGR)
    cv2.imshow("hello mask", frame)
    self.listener_keys()

def exit(self):
  """
   Program exit 
  :return:
  """
  self.video_capture.release()
  cv2.destroyAllWindows()

Finally, let's try:

if __name__ == '__main__':
  ms = DynamicStreamMaskService()
  ms.start()