Example of a method using Nginx reverse proxy to go fastdfs

  • 2020-05-15 03:43:44
  • OfStack

background

go - fastdfs is distributed file system support http 1 of the agreement, in 1 of the project, are rarely exposed directly to the address of the file system, the most by nginx software such as the generation of the past, due to our business and the scene is relatively special network environment and the public sections (public cloud) and internal (private cloud) hybrid cloud network system, public cloud is as a main export and entry and run 1 applications, such as some of the audit certification of upstream request for processing, thus reducing The Times for the processing of the private cloud to improve performance. Therefore, in a public network environment, you must use a reverse proxy to access the services provided by the private cloud. Similarly for access to file systems, how do you configure it in nginx so that external network requests can be reverse-proxied to go-fastdfs? This article will elaborate step by step.

1 a configuration

In the case of 1, as anyone familiar with nginx knows, if you need to configure a reverse proxy, just write an location context and an proxy module, and if you need a custom prefix, use an rewrite module. A simple example is as follows:


location ~ /dfs/group([0-9]) {
 proxy_pass http://localhost:8080;
 rewrite ^/dfs/(.*)$ /$1 break;
 proxy_set_header Host $host:$server_port;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

At this point, a 1-like reverse generation configuration is ok, but is it ok for go-fastdfs? Upload like go-fastdfs1 is ok, but it is not enough to use tus for breakpoint. Why? Since the tus server will return a 301 redirect and will need to carry a specified request header, a special setting of 1 is required.

Reverse generation configuration for Tus is supported

If you need to do an tus reverse generation, to support the rewriting of 301 redirection Location and to support 1 specified request header forwarding, how do you configure it? Please refer to the configuration below


location ~ /dfs1/group([0-9]) {
 access_log logs/dfs/access.log main;
 error_log logs/dfs/error.log error;
 rewrite ^/dfs1/(.*)$ /$1 break;
 proxy_pass http://localhost:8051;

 # Disable request and response buffering
 proxy_request_buffering off;
 proxy_buffering off;
 proxy_http_version 1.1;

 proxy_set_header Host $host:$server_port;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 #  if server_name Instead of a public domain name, this place can be set to ip
 proxy_set_header X-Forwarded-Host $hostname;
 proxy_set_header X-Forwarded-Proto $scheme;
 proxy_set_header Upgrade $http_upgrade;
 proxy_set_header Connection "upgrade";
 #  Because of the prefix plus rewrite , so modify the returned Location Prefix with the reverse proxy 
 proxy_redirect ~^(.*)/group([0-9])/big/upload/(.*) /dfs/group$2/big/upload/$3;
 client_max_body_size 0;
}

Note that the first configuration is proxy_redirect and client_max_body_size. The first configuration is because the redirection Location returned by tus server does not carry custom prefixes, so you need to add your own custom prefixes. The second is client_max_body_size, which means that no matter how big the file is uploaded, request too large will not be reported, and it will be directly forwarded to you. If you need to set it, please set the number greater than or equal to chunkSize. What is a chunkSize & # 63; This is the size of each block when the tus client is uploaded in blocks. Please refer to the official documents for details.

Load balancing configuration

How do you load balance uploads or downloads when the cluster server is configured? Using nginx to do reverse generation, with the upstream module can be realized, specific reference to the following configuration


upstream dfs_stream {
 server host1:port;
 server host2:port;
 ip_hash;
}

There is nothing wrong with the above configuration and 1-like load balancing. The only thing to note for 1 is to configure ip_hash. Why? Because files are uploaded in chunks when using breakpoint continuation, if not ip_hash, the first few pieces may be uploaded to A server, and the last few pieces to B server, then the file will not be complete, so please pay attention to this problem.


Related articles: