Write an example of an ES1en ES2en method using golang

0. redis communication protocol

The communication between redis's client (ES6en-ES7en) and server (ES8en-ES9en) is based on the tcp connection, and the encoding and decoding method of data transmission between them is the so-called redis communication protocol. So, as long as our ES12en-ES13en implements the parsing and encoding of this protocol, then we can complete all redis operations.

The redis protocol is designed to be very readable and easy to implement. For the specific redis communication protocol, please refer to: Communication Protocol (protocol). We will briefly repeat the implementation in the process of implementing this protocol

1. Establish the tcp connection

The communication between the redis client and server is based on the establishment of the tcp connection, so the first step is naturally to establish the connection first

package main

import (
 "flag"
 "log"
 "net"
)

var host string
var port string

func init() {
 flag.StringVar(&host, "h", "localhost", "hsot")
 flag.StringVar(&port, "p", "6379", "port")
}

func main() {
 flag.Parse()

 tcpAddr := &net.TCPAddr{IP: net.ParseIP(host), Port: port}
 conn, err := net.DialTCP("tcp", nil, tcpAddr)
 if err != nil {
 log.Println(err)
  }
  defer conn.Close()

 // to be continue
}

We can then send and receive data using conn.Read () and conn.Write ()

2. Send a request

The first byte of the sending request is "*", with the number of arguments containing the command itself followed by "\r\n". Then use "$" plus the number of parameter bytes and the end of "\r\n", followed by the end of "\r\n". "*3\r\n$3\r\ r\n$3\ nkey\r\n$7\ nliangwt\r\n"

Note:

The command itself is also sent as one of the parameters of the protocol \r\n corresponds to byte in decimal 13 10

We can use telnet to test under

wentao@bj: ~ /github.com/liangwt/redis-cli$ telnet 127.0.0.1 6379
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
*3
$3
SET
$3
key
$7
liangwt
+OK

We will ignore the reply from the server for the moment. We can see from telnet that the request protocol is very simple, so we will not introduce too much about the implementation of the request protocol, but put the code directly (the following USES string-based concatenation, just for a more intuitive demonstration, the efficiency is not high, we use ES76en.Buffer to implement in the actual code).

func MultiBulkMarshal(args ...string) string {
 var s string
 s = "*"
 s += strconv.Itoa(len(args))
 s += "\r\n"

 //  Command all parameters 
 for _, v := range args {
 s += "$"
 s += strconv.Itoa(len(v))
 s += "\r\n"
 s += v
 s += "\r\n"
 }

 return s
}

Once the commands and parameters are encoded, we can push the data to the server via ES81en.Write ()

func main() {
  // ....
 req := MultiBulkMarshal("SET", "key", "liangwt")
 _, err = conn.Write([]byte(req))
 if err != nil {
 log.Fatal(err)
 }
 // to be continue
}

Get a reply

We first achieve the server return value via tcp, which is conn.Read () mentioned above.

func main() {
  // ....
 p := make([]byte, 1024)
 _, err = conn.Read(p)
 if err != nil {
 log.Fatal(err)
 }
 // to be continue
}

4. Analyze your responses

Once we have the p we can parse the return value, and the redis server replies fall into several categories

State the reply Error response Integer reply Batch reply Multiple batch replies

We treat the first four separately as a set of 1, because they are all single-type return values

We treat the final multiple batch replies as a single group 1 because it is a hybrid type containing the previous types. And you can see that it is the same as our request protocol

Based on the above considerations, we created two functions to resolve the single type 1 and the mixed type respectively, so that when resolving a type 1 in the mixed type, we only need to call the single type resolved function

Before parsing the specific protocol, we first implement a function that reads to \r\n

func ReadLine(p []byte) ([]byte, error) {
 for i := 0; i < len(p); i++ {
 if p[i] == '\r' {
  if p[i+1] != '\n' {
  return []byte{}, errors.New("format error")
  }
  return p[0:i], nil
 }
 }
 return []byte{}, errors.New("format error")
}

Type 1 Status reply:

The status reply is a one-line string ending with "+" and "\r\n". Return value for successful SET command: "+OK\r\n"

So we determine if the first character is equal to '+' and if so, read \r\n

func SingleUnMarshal(p []byte) ([]byte, int, error) {
 var (
 result []byte
 err  error
 length int
 )
 switch p[0] {
 case '+':
 result, err = ReadLine(p[1:])
 length = len(result) + 3
 }

 return result, length, err
}

Note: When we return the actual reply content, we also return the length of the entire reply, so as to locate the parse position of the next time when multiple batch replies are parsed later

Type 2 Error reply:

The first byte of the error reply is a one-line string ending with "-", "\r\n". "-ES138en wrong of arguments for 'set' command\r\n"

Error reply and status reply are very similar, and the parsing method is the same. So we just need to add 1 case

func SingleUnMarshal(p []byte) ([]byte, int, error) {
 var (
 result []byte
 err  error
 length int
 )
 switch p[0] {
 case '+', '-':
 result, err = ReadLine(p[1:])
 length = len(result) + 3
 }
 return result, length, err
}

The third type of integer reply:

The first byte of the integer reply is ":", with the integer represented by the string in the middle, and the single-line string ending "\r\n". Return ":10\r\n" when executing LLEN mylist

The integer reply is the same as the two above, except that it returns a string representation of a decimal integer

func SingleUnMarshal(p []byte) ([]byte, int, error) {
 var (
 result []byte
 err  error
 length int
 )
 switch p[0] {
 case '+', '-', ':':
 result, err = ReadLine(p[1:])
 length = len(result) + 3
 }
 return result, length, err
}

Type 4 Batch reply:

The first byte of the batch reply is "$", followed by the string integer representing the actual reply length, followed by 1 "\r\n", followed by the actual reply data, and ending with another "\r\n". For example, the return value of the GET key command: "$7\r\nliangwt\r\n"

Therefore, the implementation of batch reply parsing:

Read line 1 to get the actual length of the reply Converts the length of a string type to the corresponding decimal integer Read the corresponding length from the beginning of line 2

But for some nonexistent key, the batch reply will use the special value -1 as the length of the reply, at which point we don't need to read the actual reply down. For example, GET NOT_EXIST_KEY returns a value of "$-1", so we need to determine for this special case that the function returns an empty object (nil) instead of a null value ("").

func SingleUnMarshal(p []byte) ([]byte, int, error) {
 // ....
 case '$':
 n, err := ReadLine(p[1:])
 if err != nil {
  return []byte{}, 0, err
 }
 l, err := strconv.Atoi(string(n))
 if err != nil {
  return []byte{}, 0, err
 }
 if l == -1 {
  return nil, 0, nil
 }
 // +3  The reason why  $ \r \n 3 A character 
 result = p[len(n)+3 : len(n)+3+l]
 length = len(n) + 5 + l
 }
 return result, length, err
}

Think about:

Why did redis tell you the number of bytes in advance and then read down the specified length, rather than just reading line 2 until \r\n?

The answer is obvious: this way, redis can read the return value without being affected by the specific return content. In the case of line-by-line reading, any use of a delimiter in the content may cause redis to parse the specific content using the delimiter in the content as the end of the time, resulting in a parse error.

Consider this situation: we SET key "liang\r\nwt", so when we GET key, the server returns "$9\r\nliang\r\ r\n" completely evasive the influence of \r\n in value

The fifth type of multiple batch reply:

Multiple batch replies are arrays of replies with the first byte of "*" followed by an integer value of 1 string that records the number of replies contained in multiple batch replies, followed by 1 "\r\n". LRANGE mylist 0-1 returns: "*3\r\ r\n3\r\ r\n2\ n$1\ n1".

Therefore, the implementation of multiple batch reply parsing:

Parses line 1 data for the number of replies of string type Converts the length of a string type to the corresponding decimal integer Parse one by one according to a single reply, and 1 co-parse to the number obtained above

Here we use the byte length length returned from a single parse, from which we can easily know that the starting position of the next single parse is the previous position +length

When analyzing multiple batch replies, two points should be noted:

1. Multiple batch replies can also be blank (empty). For example, execute LRANGE NOT_EXIST_KEY 0-1 server return value "*0\r\n". The client should return an empty array [][]byte

Second, multiple batch replies can also be content-free (null multi bulk reply). For example, execute BLPOP key 1 server return value "*-1\r\n". The client should return nil

wentao@bj: ~ /github.com/liangwt/redis-cli$ telnet 127.0.0.1 6379
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
*3
$3
SET
$3
key
$7
liangwt
+OK

0

5. Command line mode

One available redis-ES279en is naturally an interactive one, with the user entering instructions and then outputting the return value. In go we can get a similar interactive command line using the following code

wentao@bj: ~ /github.com/liangwt/redis-cli$ telnet 127.0.0.1 6379
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
*3
$3
SET
$3
key
$7
liangwt
+OK

1

We can do this by running the above code

wentao@bj: ~ /github.com/liangwt/redis-cli$ telnet 127.0.0.1 6379
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
*3
$3
SET
$3
key
$7
liangwt
+OK

2

The whole redis-cli can be completed by combining our redis sending request and parsing request

func main() {
  // ....
 for {
 fmt.Printf("%s:%d>", host, port)

 //  Gets the input commands and parameters 
 bio := bufio.NewReader(os.Stdin)
 input, err := bio.ReadString('\n')
 if err != nil {
  log.Fatal(err)
 }
 fields := strings.Fields(input)

 //  Coded send request 
 req := MultiBulkMarshal(fields...)

 //  Send the request 
 _, err = conn.Write([]byte(req))
 if err != nil {
  log.Fatal(err)
 }

 //  Read back 
 p := make([]byte, 1024)
 _, err = conn.Read(p)
 if err != nil {
  log.Fatal(err)
 }

 //  Parse the returned contents 
 if p[0] == '*' {
  result, err := MultiUnMarsh(p)
 } else {
  result, _, err := SingleUnMarshal(p)
 }

  }
  // ....
}

6. Summary

So far our cli program is complete, but there are still many imperfections. But the core redis protocol parsing is complete, and we can use this parsing to complete any interaction between cli and the server

For a more detailed implementation of redis-ES302en refer to my github: A Simaple redis ES307en-ES308en