A simple Golang implementation of the HTTP Proxy method

Recently, the previous Linux has been largely abandoned due to the replacement of Mac, but my SS agent is still in use. SS agent as we all know, a very NB socks agent tool, but because it is Socks, it is not convenient to use HTTP agent.

In the past, under Linux, one Privoxy would be installed to convert socks agent to HTTP agent, which is convenient to boot up. But Mac using Brew installation Privoxy is difficult to use, plus before 1 had an idea, 1 software to handle socks and HTTP agents, so there is no need to install a separate software to do the conversion.

Think about it, before basically did not do too much network programming, recently also happened to study Go, just practice.

Here we mainly discuss the tunnel connection established by using CONNECT method in HTTP / 1.1 protocol, and the realization of HTTP Proxy. The advantage of this proxy is that it does not need to know the data requested by the client, but only needs to be forwarded intact. It is very convenient to handle the request of HTTPS, and the proxy can be implemented without parsing its content.

Start agent listening

To do an HTTP Proxy, we need to start a server and listen on a port to receive requests from the client. Golang gave us a powerful net package to use, and it was very convenient for us to start a proxy server listening.

  l, err := net.Listen("tcp", ":8080")
  if err != nil {
    log.Panic(err)
  }

Above agent we realized a server listening on port 8080, we did not write ip address here, default to listen on all ip addresses. If you only want to be native, you can use 127.0.0.1:8080 so that the machine can't access your proxy server.

Listen to receive proxy requests

Once the proxy server is started, we can begin to fail the proxy request, and with the request, we can proceed to the next step.

  for {
    client, err := l.Accept()
    if err != nil {
      log.Panic(err)
    }

    go handleClientRequest(client)
  }

The Accept method of the Listener interface will accept connection data sent by the client. This is a blocking method. If the client does not receive connection data, it will block and wait. The received connection data is immediately handed over to the handleClientRequest method for processing. The purpose of using one go keyword to open one goroutine is not to block the client's receiving, and the proxy server can receive the next connection request immediately.

Parse the request to get the IP and port to access

With the proxy request from the client, we also have to extract the IP and port of the remote host that the client wants to access from the request, so that our proxy server can establish a connection to the remote host and proxy forward.

The header information of HTTP protocol contains the host name (IP) and port information we need, and it is clear text. The protocol is very standard, similar to:

CONNECT www.google.com:443 HTTP/1.1
Host: www.google.com:443
Proxy-Connection: keep-alive
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36

You can see that the information we need on line 1 is separated by Spaces. In Part 1, CONNECT is the request method, here is CONNECT, in addition to GET, POST, etc., which are standard methods of the HTTP protocol.

The second part is URL. The request of https is only host and port. The request of http is one completed url.

Part 3 is the protocol and version of HTTP, which we won't pay much attention to.

The above is a request for https, let's take a look at http's:

GET http://www.flysnow.org/ HTTP/1.1
Host: www.flysnow.org
Proxy-Connection: keep-alive
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36

As you can see, htt has no port number (default is 80); schame?? http://.

With this analysis in place, we can now retrieve the requested url and method information from the HTTP header.

  var b [1024]byte
  n, err := client.Read(b[:])
  if err != nil {
    log.Println(err)
    return
  }
  var method, host, address string
  fmt.Sscanf(string(b[:bytes.IndexByte(b[:], '\n')]), "%s%s", &method, &host)
  hostPortURL, err := url.Parse(host)
  if err != nil {
    log.Println(err)
    return
  
  }

Then we need to take a step to parse url to get the remote server information we need

  if hostPortURL.Opaque == "443" { //https access 
    address = hostPortURL.Scheme + ":443"
  } else { //http access 
    if strings.Index(hostPortURL.Host, ":") == -1 { //host Without ports,   The default 80
      address = hostPortURL.Host + ":80"
    } else {
      address = hostPortURL.Host
    }
  }

This completes the retrieval of the information to request from the server, which may be in one of the following formats

ip:port
hostname:port
domainname:port

It could be ip (v4orv6), it could be hostname (Intranet), it could be domain name (dns parse)

The proxy server and the remote server establish a connection

With the remote server information, you can dial-up to establish a connection, there is a connection, you can communicate.

  // Got the requested one host and port Start dialing 
  server, err := net.Dial("tcp", address)
  if err != nil {
    log.Println(err)
    return
  }

Data forwarding

After dialing successfully, the data agent transfer is ready

if method == "CONNECT" {
    fmt.Fprint(client, "HTTP/1.1 200 Connection established\r\n")
  } else {
    server.Write(b[:n])
  }
  // For forwarding 
  go io.Copy(server, client)
  io.Copy(client, server)

There is a separate response to the CONNECT method, where the client says to establish a connection and the proxy server replies that it is established before it can request access like HTTP1.

Run on the foreign VPS

Here, all of our proxy server development is completed, the following is the complete source code:

package main

import (
  "bytes"
  "fmt"
  "io"
  "log"
  "net"
  "net/url"
  "strings"
)

func main() {
  log.SetFlags(log.LstdFlags|log.Lshortfile)
  l, err := net.Listen("tcp", ":8081")
  if err != nil {
    log.Panic(err)
  }

  for {
    client, err := l.Accept()
    if err != nil {
      log.Panic(err)
    }

    go handleClientRequest(client)
  }
}

func handleClientRequest(client net.Conn) {
  if client == nil {
    return
  }
  defer client.Close()

  var b [1024]byte
  n, err := client.Read(b[:])
  if err != nil {
    log.Println(err)
    return
  }
  var method, host, address string
  fmt.Sscanf(string(b[:bytes.IndexByte(b[:], '\n')]), "%s%s", &method, &host)
  hostPortURL, err := url.Parse(host)
  if err != nil {
    log.Println(err)
    return
  }

  if hostPortURL.Opaque == "443" { //https access 
    address = hostPortURL.Scheme + ":443"
  } else { //http access 
    if strings.Index(hostPortURL.Host, ":") == -1 { //host Without ports,   The default 80
      address = hostPortURL.Host + ":80"
    } else {
      address = hostPortURL.Host
    }
  }

  // Got the requested one host and port Start dialing 
  server, err := net.Dial("tcp", address)
  if err != nil {
    log.Println(err)
    return
  }
  if method == "CONNECT" {
    fmt.Fprint(client, "HTTP/1.1 200 Connection established\r\n")
  } else {
    server.Write(b[:n])
  }
  // For forwarding 
  go io.Copy(server, client)
  io.Copy(client, server)
}

Compile the source code, put it on your VPS abroad, configure the HTTP agent on your own machine, and you can access it everywhere freely.