Implement a simple FastCGI server instance using node. js

  • 2020-03-30 03:15:45
  • OfStack

This article is one of my recent ideas on learning node. js.

The HTTP server for node. js

Node.js makes it very easy to implement an HTTP service. The simplest example is the example of an official website:


var http = require('http');
http.createServer(function (req, res) {
    res.writeHead(200, {'Content-Type': 'text/plain'});
    res.end('Hello Worldn');
}).listen(1337, '127.0.0.1');

This quickly builds a web service that listens for all HTTP requests on port 1337.
However, in a real production environment, we seldom directly use node. js as the most front-end web server for users, for the following reasons:

1. Due to the single thread feature of node. js, the guarantee of its robustness is quite demanding for developers.
2. There may be other HTTP services on the server that already occupy port 80, and a web service that is not port 80 is clearly not user friendly.
3.Node.js does not have much advantage in file IO processing, for example, as a regular website may need to respond to file resources such as images at the same time.
4. Distributed load scenarios are also a challenge.

Therefore, using node.js as a web service is more likely to be a game server interface or similar scenarios, mostly dealing with services that do not require direct user access and are only for data exchange.

Node.js web service based on Nginx as the front-end machine

For the above reasons, if it is a web-like product built with node. js, the normal way to use it is to place another mature HTTP server in the front of the web service of node. js, such as Nginx, which is most commonly used.
Then use Nginx as a reverse proxy to access the node.js-based web service. Such as:


server{
    listen 80;
    server_name yekai.me;
    root /home/andy/wwwroot/yekai;
    location / {
        proxy_pass http://127.0.0.1:1337;
    }
    location ~ .(gif|jpg|png|swf|ico|css|js)$ {
        root /home/andy/wwwroot/yekai/static;
    }
}

This is a better solution to the above several problems.

Communicate using FastCGI protocol

However, there are some disadvantages of the above approach to agency.
One possible scenario is that you need to control direct HTTP access to the web service behind node. js. However, you can also use your own services or rely on firewall blocking.
Another reason is that the proxy approach is, after all, a network application layer solution, and it is not very convenient to directly retrieve and process data that interacts with client HTTP, such as keep-alive, trunk, and even cookies. Of course, this is also related to the capabilities and functionality of the proxy server itself.
So, I was thinking of trying another approach, and the first one that came to mind was FastCGI, which is now commonly used in PHP web applications.

What is a FastCGI

Fast Common Gateway Interface (FastCGI) is a protocol that allows interactive programs to communicate with Web servers.

FastCGI web application background is used as a cgi alternatives, one of the most obvious feature is a FastCGI service process can be used to deal with a series of requests, the web server will reduce the environment variables and the page requests through a socket such as FastCGI process and the web server, and the connection available Domain Unix socket or a TCP/IP connection. Refer to the Wikipedia entry for more background information.

Node.js FastCGI implementation

In theory, we just need to create a FastCGI process using node. js and then specify that Nginx's listening request is sent to the process. Since both Nginx and node. js are based on event-driven service models, "in theory" should be a natural solution. So let's do that ourselves.
Net module in node. js is just available to establish a socket service, in order to facilitate the use of Unix socket.
The configuration of Nginx side is slightly modified:


...
location / {
    fastcgi_pass   unix:/tmp/node_fcgi.sock;
}
...

New file node_fcgi.js, as follows:

var net = require('net');
var server = net.createServer();
server.listen('/tmp/node_fcgi.sock');
server.on('connection', function(sock){
    console.log('connection');
    sock.on('data', function(data){
        console.log(data);
    });
});


Then run (because of permissions, make sure that the Nginx and node scripts run on the same user or with mutual permissions, or you will encounter permissions problems reading and writing sock files) :

The node node_fcgi. Js

In the browser access, we can see that the terminal running the node script normally receives the data content, such as this:


connection
< Buffer 01 01 00 01 00 08 00 00 00 01 00 00 00 00 00 00 01 04 00 01 01 87 01...>

This proves that our theoretical foundation has achieved the first step. The next step is to figure out how the contents of the buffer are parsed.


FastCGI protocol basis

A FastCGI record consists of a fixed-length prefix followed by a variable number of content and padding bytes. The record structure is as follows:


typedef struct {
    unsigned char version;
    unsigned char type;
    unsigned char requestIdB1;
    unsigned char requestIdB0;
    unsigned char contentLengthB1;
    unsigned char contentLengthB0;
    unsigned char paddingLength;
    unsigned char reserved;
    unsigned char contentData[contentLength];
    unsigned char paddingData[paddingLength];
} FCGI_Record;

Version: FastCGI protocol version, now default to 1
Type: record type, which can be thought of as different states, as described later
RequestId: requestId, return to the corresponding, if not multiplexed concurrent case, just use 1
ContentLength: length of content. The maximum length here is 65535
PaddingLength: fill length to fill an integer multiple of the full 8 bytes of long data, mainly for more efficient processing of data that remains aligned, mainly for performance reasons
Reserved: reserved bytes for later expansion
ContentData: real contentData, more on that in a moment
PaddingData. It's all 0, so just ignore it.

Concrete structure and document please refer to the website (http://www.fastcgi.com/devkit/doc/fcgi-spec.html#S3.3).


Request part

It seems simple enough to parse the data once you get it. However, there is a pitfall, which is that the structure of the data unit (record) is defined here, not the structure of the entire buffer, which consists of record by record. This may not be easy to understand at first for those of us who are used to front-end development, but it is the basis for understanding the FastCGI protocol, and we will see more examples later.
So, we need to parse out one record by one, and distinguish the records according to the type we got earlier. Here is a simple function to get all records:


function getRcds(data, cb){
    var rcds = [],
        start = 0,
        length = data.length;
    return function (){
        if(start >= length){
            cb && cb(rcds);
            rcds = null;
            return;
        }
        var end = start + 8,
            header = data.slice(start, end),
            version = header[0],
            type    = header[1],
            requestId = (header[2] << 8) + header[3],
            contentLength = (header[4] << 8) + header[5],
            paddingLength = header[6];
        start = end + contentLength + paddingLength;
        var body = contentLength ? data.slice(end, contentLength) : null;
        rcds.push([type, body, requestId]);
        return arguments.callee();
    }
}
// use 
sock.on('data', function(data){
    getRcds(data, function(rcds){
    })();
}

Note that this is a simple process, but if there are some complicated situations such as uploading files that this function is not suitable for, I'll do it for the sake of simplicity. It also ignores the requestId parameter, which cannot be ignored in the case of multiplexing and would require much more complex processing.
You can then process the different records by type. The definition of type is as follows:


#define FCGI_BEGIN_REQUEST       1
#define FCGI_ABORT_REQUEST       2
#define FCGI_END_REQUEST         3
#define FCGI_PARAMS              4
#define FCGI_STDIN               5
#define FCGI_STDOUT              6
#define FCGI_STDERR              7
#define FCGI_DATA                8
#define FCGI_GET_VALUES          9
#define FCGI_GET_VALUES_RESULT  10
#define FCGI_UNKNOWN_TYPE       11
#define FCGI_MAXTYPE (FCGI_UNKNOWN_TYPE)

Next, the real data can be parsed according to the type of the record. Next, I will only take the most commonly used FCGI_PARAMS, FCGI_GET_VALUES, and FCGI_GET_VALUES_RESULT to explain. Fortunately, their parsing methods are consistent. The parsing of other type records has its own different rules, you can refer to the definition of the specification, I will not go into details here.
FCGI_PARAMS, FCGI_GET_VALUES, FCGI_GET_VALUES_RESULT are all data of "encoding name-value" type. The standard format is: the length of the name, followed by the length of the value, followed by the name, followed by the value. The high digit of the first byte of length indicates how the length is encoded. The high digit of 0 means the encoding of one byte, and the high digit of 1 means the encoding of four bytes. Let's look at an example of synthesis, such as the case of long names and short values:


typedef struct {
    unsigned char nameLengthB3;  
    unsigned char nameLengthB2;
    unsigned char nameLengthB1;
    unsigned char nameLengthB0;
    unsigned char valueLengthB0; 
    unsigned char nameData[nameLength
            ((B3 & 0x7f) << 24) + (B2 << 16) + (B1 << 8) + B0];
    unsigned char valueData[valueLength];
} FCGI_NameValuePair41;

Example of corresponding js method:


function parseParams(body){
    var j = 0,
        params = {},
        length = body.length;
    while(j < length){
        var name,
            value,
            nameLength,
            valueLength;
        if(body[j] >> 7 == 1){
            nameLength = ((body[j++] & 0x7f) << 24) + (body[j++] << 16) + (body[j++] << 8) + body[j++];
        } else {
            nameLength = body[j++];
        }
        if(body[j] >> 7 == 1){
            valueLength = ((body[j++] & 0x7f) << 24) + (body[j++] << 16) + (body[j++] << 8) + body[j++];
        } else {
            valueLength = body[j++];
        }
        var ret = body.asciiSlice(j, j + nameLength + valueLength);
        name = ret.substring(0, nameLength);
        value = ret.substring(nameLength);
        params[name] = value;
        j += (nameLength + valueLength);
    }
    return params;
}

This provides a simple way to get various parameters and environment variables. Complete the previous code to demonstrate how we get the client IP:


sock.on('data', function(data){
    getRcds(data, function(rcds){
        for(var i = 0, l = rcds.length; i < l; i++){
            var bodyData = rcds[i],
                type = bodyData[0],
                body = bodyData[1];
            if(body && (type === TYPES.FCGI_PARAMS || type === TYPES.FCGI_GET_VALUES || type === TYPES.FCGI_GET_VALUES_RESULT)){
                    var params = parseParams(body);
                    console.log(params.REMOTE_ADDR);
                }
        }
    })();
}

Now that we know the basics of the FastCGI request section, let's move on to the response section and finally complete a simple echo reply service.

The response part

The response part is relatively simple, in the simplest case just sending two records is FCGI_STDOUT and FCGI_END_REQUEST.
The content of specific record entity is not redundant, look at the code directly:


var res = (function(){
    var MaxLength = Math.pow(2, 16);
    function buffer0(len){
        return new Buffer((new Array(len + 1)).join('u0000'));
    };
    function writeStdout(data){
        var rcdStdoutHd = new Buffer(8),
            contendLength = data.length,
            paddingLength = 8 - contendLength % 8;
        rcdStdoutHd[0] = 1;
        rcdStdoutHd[1] = TYPES.FCGI_STDOUT;
        rcdStdoutHd[2] = 0;
        rcdStdoutHd[3] = 1;
        rcdStdoutHd[4] = contendLength >> 8;
        rcdStdoutHd[5] = contendLength;
        rcdStdoutHd[6] = paddingLength;
        rcdStdoutHd[7] = 0;
        return Buffer.concat([rcdStdoutHd, data, buffer0(paddingLength)]);
    };
    function writeHttpHead(){
        return writeStdout(new Buffer("HTTP/1.1 200 OKrnContent-Type:text/html; charset=utf-8rnConnection: closernrn"));
    }
    function writeHttpBody(bodyStr){
        var bodyBuffer = [],
            body = new Buffer(bodyStr);
        for(var i = 0, l = body.length; i < l; i += MaxLength + 1){
            bodyBuffer.push(writeStdout(body.slice(i, i + MaxLength)));
        }
        return Buffer.concat(bodyBuffer);
    }
    function writeEnd(){
        var rcdEndHd = new Buffer(8);
        rcdEndHd[0] = 1;
        rcdEndHd[1] = TYPES.FCGI_END_REQUEST;
        rcdEndHd[2] = 0;
        rcdEndHd[3] = 1;
        rcdEndHd[4] = 0;
        rcdEndHd[5] = 8;
        rcdEndHd[6] = 0;
        rcdEndHd[7] = 0;
        return Buffer.concat([rcdEndHd, buffer0(8)]);
    }
    return function(data){
        return Buffer.concat([writeHttpHead(), writeHttpBody(data), writeEnd()]);
    };
})();

In the simplest case, you can send a complete response. Modify our final code:


var visitors = 0;
server.on('connection', function(sock){
    visitors++;
    sock.on('data', function(data){
        ...
        var querys = querystring.parse(params.QUERY_STRING);
            var ret = res(' Welcome you, ' + (querys.name || ' Dear friend ') + ' ! You're on this site ' + visitors + ' A user oh ~');
            sock.write(ret);
            ret = null;
            sock.end();
        ...
    });

Open browser access: http://domain/? Name =yekai, you will see something like "welcome, yekai! You are the 7th user of this site.
So far, we have successfully implemented a simple FastCGI service using node. js. If we need to use it as a real service, then we just need to refine our logic against the protocol specification.


Contrast test

Afterword.


Related articles: