Implement the server restart method in the Go program

  • 2020-05-30 20:23:34
  • OfStack

Go is designed as a background language, and it is often used in back-end applications as well. Server-side programs are the most common software products in the GO language. My problem here is how to cleanly upgrade a running server application.
Goal:

Do not close an existing connection: for example, we do not want to close a deployed running program. But they want to upgrade at any time without restrictions. An socket connection is always responsive to a user request: closing socket at any time may cause the user to return a 'connection denied' message, which is undesirable. The new process needs to be able to start and replace the old one.

The principle of

In Unix-based operating systems, signal(signals) is a common way to interact with long-running processes.

SIGTERM: gracefully stop the process SIGHUP: restart/reload the process (e.g. nginx, sshd, apache)

If you receive an SIGHUP signal, the following steps are required to gracefully restart the process:

The server rejects the new connection request, but maintains the existing connection. Enable the new version of the process socket is "handed over" to the new process, which begins accepting new connection requests Stop the old process as soon as it has been processed.

Stop accepting connection requests

Common features of server applications: holding 1 loop to accept connection requests:

 for {
      conn, err := listener.Accept()
      // Handle connection
    }

The easiest way to get out of this loop is to set a timeout on the socket listener. When listener.SetTimeout (time.Now ()) is called, listener.Accept () immediately returns an timeout err, which you can capture and process:

 for {
      conn, err := listener.Accept()
      if err != nil {
        if nerr, ok := err.(net.Err); ok && nerr.Timeout() {
           fmt.Println( " Stop accepting connections " )
           return
        }
      }
    }

Note that this is different from closing listener. The process is still listening on the server port, but the connection requests are queued on the operating system's network stack for one process to accept them.
Start a new process

Go provides a primitive type ForkExec to generate a new process. You can share some messages with this new process, such as file descriptors or environment parameters.

 execSpec := &syscall.ProcAttr{
      Env:   os.Environ(),
      Files: []uintptr{os.Stdin.Fd(), os.Stdout.Fd(), os.Stderr.Fd()},
    }
    fork, err := syscall.ForkExec(os.Args[0], os.Args, execSpec)
    [ ... ]

You will notice that this process starts a new process using the exact same parameter os.Args.
Send socket to the child process and restore it

As you saw earlier, you can pass the file descriptor to the new process, which requires some UNIX magic. We can send socket to the new process so that the new process can use it and receive and wait for a new connection.

But the fork-execed process needs to know that it must get the socket from the file instead of creating a new one (some may already be in use, since we haven't disconnected the existing listeners yet). You can do it any way you want, most often through environment variables or command-line flags.

listenerFile, err := listener.File()
    if err != nil {
      log.Fatalln("Fail to get socket file descriptor:", err)
    }
    listenerFd := listenerFile.Fd()
    
    // Set a flag for the new process start process
    os.Setenv("_GRACEFUL_RESTART", "true")
    
    execSpec := &syscall.ProcAttr{
      Env:   os.Environ(),
      Files: []uintptr{os.Stdin.Fd(), os.Stdout.Fd(), os.Stderr.Fd(), listenerFd},
    }
    // Fork exec the new version of your server
    fork, err := syscall.ForkExec(os.Args[0], os.Args, execSpec)

Then at the beginning of the program:

 var listener *net.TCPListener
    if os.Getenv("_GRACEFUL_RESTART") == "true" {
      // The second argument should be the filename of the file descriptor
      // however, a socker is not a named file but we should fit the interface
      // of the os.NewFile function.
      file := os.NewFile(3, "")
      listener, err := net.FileListener(file)
      if err != nil {
        // handle
      }
      var bool ok
      listener, ok = listener.(*net.TCPListener)
      if !ok {
        // handle
      }
    } else {
      listener, err = newListenerWithPort(12345)
    }

The file description was not randomly selected as 3, because the slice of uintptr has been sent to fork, and the monitor obtained the index 3. Watch out for implicit declarations.
Last step, wait for the old service connection to stop

At this point, we have passed it to another process that is running correctly, and the last thing to do with the old server is to wait for its connection to close. Since the standard library provides the sync.WaitGroup structure, it is easy to implement this functionality with go.

One connection at a time, add 1 to WaitGroup, and then, when it's done, we subtract 1 from the counter:

 for {
      conn, err := listener.Accept()
    
      wg.Add(1)
      go func() {
        handle(conn)
        wg.Done()
      }()
    }

As for waiting for the end of the connection, you only need wg.Wait (), since there is no new connection, we are waiting for wg.Done () to have been called by all running handler.
Bonus: don't wait indefinitely for a given amount of time

 timeout := time.NewTimer(time.Minute)
    wait := make(chan struct{})
    go func() {
      wg.Wait()
      wait <- struct{}{}
    }()
    
    select {
    case <-timeout.C:
      return WaitTimeoutError
    case <-wait:
      return nil
    }

Complete example

Code snippet of the article is extracted from the complete example: https: / / github com/Scalingo/go - graceful restart -- example
conclusion

socket delivery with ForkExec usage is indeed an effective way to do a non-disruptive update process. At the maximum time, the new connection will wait a few milliseconds -- for the service to start and restore socket, but this time is very short.


Related articles: