SpringBoot integration FastDFS + Nginx integration method of anti theft chain based on Token

Why use SpringBoot?

SpringBoot is a new framework provided by Pivotal team, which is designed to simplify the initial construction and development process of new Spring applications. The framework uses a specific way to configure, so that developers no longer need to define boilerplate configurations. In this way, Spring Boot strives to be a leader in the burgeoning field of rapid application development (rapid application development).

Create a stand-alone Spring application Embedded Tomcat, no need to deploy WAR files Simplify Maven configuration Automatically configure Spring Provides production-ready features such as metrics, health checks, and external configuration Absolutely no code generation and no configuration requirements for XML

Why use Nginx?

Overview

Nginx (engine x) is an open source www service and proxy service software that supports high concurrency. Nginx was developed by Russian Igor Sysoev, and was originally applied to large websites in Russia (www. rambler. ru). Later, the author opened the source code in the form of BSD-like license for global use. In terms of functional application, Nginx is not only an excellent Web service software, but also has the functions of reverse proxy load balancing and caching. In reverse proxy load balancing, it is similar to your professional proxy software such as LVS load balancing and HAProxy. Nginx is more convenient and simple to deploy. In terms of cache service function, there are professional cache service software similar to Squid. Nginx can run in UNIX, Linux, MS Windows Server, Mac OS X Server, Solaris and other operating systems.

Important Characteristics of Nginx

High-speed node concurrent access and cache for static resources. Reverse proxy acceleration can be used, and data caching can be performed. It has simple load balancing, node health check and fault tolerance functions. Support cache acceleration for remote Fast CGI services. Support Fast CGI, Uwsgi, SCGI, Memcached Server acceleration and caching. Support SSL, TLS and SNI. Has a modular architecture. Filters include gzip compression, ranges support, chunked response, XSLT, SSL, and image scaling. In the SSL filter, there are multiple SSL pages, which can be processed in parallel if processed via Fast CGI or reverse proxy.

WWW service features of Nginx

Supports virtual hosting configuration based on domain name, port and IP. Support KeepAlived and piplined connections. It can be configured and managed simply, conveniently and flexibly. Support to modify Nginx configuration, and when the code goes online, it can restart smoothly without interrupting business access. You can customize access log formats, temporarily buffer write log operations, fast log polling, and process logs via rsyslog. The Nginx process can be controlled by signals. Support 3xx-5xxHTTP status code redirection. Support rewrite module, support URI rewriting and regular expression matching. Support access control based on client IP address and HTTP basic authentication. Support special HTTP request methods such as PUT, DELETE, MKCOL, COPY, MOVE, etc. Support FLV stream and MP4 stream technology product applications. Support HTTP response rate limit. Support concurrent connection or request restriction with 1IP address. Support mail service agent. Supports high concurrency and can support millions of concurrent connections. With less resource consumption, under 30,000 concurrent connections, threads that can open 10 nginx consume less than 200MB in memory. Can do HTTP reverse proxy and accelerated cache, and load balancing function, built-in RS node server health check function, discounted but can be with professional HAProxy or LVS function. It has the caching function of professional caching software such as Squid. Support asynchronous network I/O event model epoll (Linux2.6 +).

Main Enterprise Applications of Nginx Software

As Web service software. Use Nginx to run static data such as HTML, JS, CSS, small pictures, etc. (similar to Lighttpd). Run dynamic programs such as PHP in conjunction with Fast and CGI (for example, using fastcgi_pass mode). Nginx combined with Tomcat/Resin supports Java dynamic programs (commonly used proxy_pass). Reverse Proxy or Load Balancing Service (Nginx has supported TCP's proxy since 1.9. 0). Front-end business data caching service.

Performance comparison of Web service application products

For accessing static data, when dealing with small files (less than 1MB), Nginx and Lighttpd have more advantages than Apache, Nginx has obvious advantages in dealing with small files, and Lighttpd has the strongest synthesis. Access to dynamic data: There is not much difference among the three, and Apache has more advantages, because the ability to process dynamic data lies in the service ability of PHP (Java) and back-end database, that is to say, the bottleneck is not on Web server. 1 The reference value for concurrent connections supported by the normal PHP engine is 30001000. Concurrent connection reference for Java engine and database 3001500.

Why is the performance of Nginx higher than that of Apache?

Nginx uses the latest versions of eepoll (Linux 2.6 kernel) and kqueue (FreeBSD) asynchronous network I/O model, while Apache uses the traditional select model. At present, epoll model is adopted by Squid and Memcached software which can withstand high concurrent access under Linux. The select network I/O model used by Apache is relatively low when dealing with a large number of connected reads and writes.

How to use Web server correctly.

Static service: If it is a high concurrency scenario, try to use Nginx or Lighttpd, and Nginx is preferred for the two. Dynamic business: In theory, Nginx and Apache can be used, and Nginx is recommended. In order to avoid software diversification of the same business service and increase maintenance cost, Nginx can be used as front-end agent for dynamic business, and then forwarded to other servers for processing according to page elements or directories. Nginx is used for both dynamic and static services.

For deployment, no more repetitions. If necessary, please go to "FastDFS Distributed File Cluster of Java Advanced Architecture":

Creating a project using IDEA scenario launcher

Create the Maven project and modify the POM. xml file to add the following dependencies:

<dependencies>
  <!-- SpringBoot Automatically configure dependent dependencies of  -->
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-autoconfigure</artifactId>
    <version>1.5.20.RELEASE</version>
  </dependency>

  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-configuration-processor</artifactId>
    <version>1.5.20.RELEASE</version>
  </dependency>
  <!--  Log-related dependencies  -->
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-logging</artifactId>
    <version>1.5.20.RELEASE</version>
  </dependency>
  <!--  Object pool dependent dependencies  -->
  <dependency>
    <groupId>org.apache.commons</groupId>
    <artifactId>commons-pool2</artifactId>
    <version>2.6.0</version>
  </dependency>

</dependencies>

Create the necessary packages

annotation: Store related annotations autoconfiguation: Storing Automatic Configuration Classes factory: Store factory classes properties: Storing configuration parameter classes service: Hosting service classes

1 In general, SpringBoot will provide the corresponding @ EnableXxx annotation to open a function on the main startup class of the application:

// EnableFastdfsClient.java
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Import(FastdfsAutoConfiguration.class)
@Documented
public @interface EnableFastdfsClient {
}

Here are the related auto-configuration classes:

// FastdfsAutoConfiguration.java
@Configuration
@EnableConfigurationProperties(FastdfsProperties.class)
public class FastdfsAutoConfiguration {
  @Autowired
  private FastdfsProperties fastdfsProperties;

  @Bean
  @ConditionalOnMissingBean(FastdfsClientService.class)
  public FastdfsClientService fastdfsClientService() throws Exception {
    return new FastdfsClientService(fastdfsProperties);
  }
}

Create related factory classes:

// StorageClientFactory.java
//  Factory class for creating connection objects 
public class StorageClientFactory implements PooledObjectFactory<StorageClient> {

  @Override
  public PooledObject<StorageClient> makeObject() throws Exception {
    TrackerClient client = new TrackerClient();
    TrackerServer server = client.getConnection();
    return new DefaultPooledObject<>(new StorageClient(server, null));
  }

  @Override
  public void destroyObject(PooledObject<StorageClient> p) throws Exception {
    p.getObject().getTrackerServer().close();
  }

  @Override
  public boolean validateObject(PooledObject<StorageClient> p) {
    return false;
  }

  @Override
  public void activateObject(PooledObject<StorageClient> p) throws Exception {

  }

  @Override
  public void passivateObject(PooledObject<StorageClient> p) throws Exception {

  }
}

The Properties class is used to map application. properties or application. yml configuration files:

// FastdfsProperties.java
@ConfigurationProperties(prefix = "fastdfs")
public class FastdfsProperties {
  //  Connection timeout 
  //  Network timeout 
  //  Character set coding 
  //  Whether to use Token
  // Token Encryption key 
  //  Tracker IP Address, multiple separated by semicolons 
  //  Maximum number of connection objects in the connection pool 
  //  Maximum number of free objects in connection pool 
  //  Minimum number of free objects in the connection pool 
  // Nginx Server IP Multiple are divided by semicolons 
  //  Tolerable wait time when getting connection objects ( Milliseconds )
  private String connectTimeout = "5";
  private String networkTimeout = "30";
  private String charset = "UTF-8";
  private String httpAntiStealToken = "false";
  private String httpSecretKey = "";
  private String httpTrackerHttpPort = "";
  private String trackerServers = "";
  private String connectionPoolMaxTotal = "18";
  private String connectionPoolMaxIdle = "18";
  private String connectionPoolMinIdle = "2";
  private String nginxServers = "";

  //  You need to create the associated Setter And Getter Method 
}

To encapsulate methods in the Service class, here are just three commonly used methods:

// FastdfsClientSerivce.java
public class FastdfsClientService {
  // SpringBoot Loaded configuration file 
  //  Connection pool configuration item 
  //  Configuration entry after conversion 
  //  Connection pool 
  // Nginx Server address 
  private FastdfsProperties fdfsProp;
  private GenericObjectPoolConfig config;
  private Properties prop;
  private GenericObjectPool<StorageClient> pool;
  private String[] nginxServers;
  private Logger logger;

  public FastdfsClientService(FastdfsProperties fdfsProp) throws Exception {
    this.fdfsProp = fdfsProp;
    this.logger = LoggerFactory.getLogger(getClass());
    init();
    create();
    info();
  }

  /**
   *  Initialize the global client 
   */
  private void init() throws Exception {
    this.prop = new Properties();
    this.logger.info("FastDFS: reading config file...");
    this.logger.info("FastDFS: fastdfs.connect_timeout_in_seconds=" + this.fdfsProp.getConnectTimeout());
    this.logger.info("FastDFS: fastdfs.network_timeout_in_seconds=" + this.fdfsProp.getNetworkTimeout());
    this.logger.info("FastDFS: fastdfs.charset=" + this.fdfsProp.getCharset());
    this.logger.info("FastDFS: fastdfs.http_anti_steal_token=" + this.fdfsProp.getHttpAntiStealToken());
    this.logger.info("FastDFS: fastdfs.http_secret_key=" + this.fdfsProp.getHttpSecretKey());
    this.logger.info("FastDFS: fastdfs.http_tracker_http_port=" + this.fdfsProp.getHttpTrackerHttpPort());
    this.logger.info("FastDFS: fastdfs.tracker_servers=" + this.fdfsProp.getTrackerServers());
    this.logger.info("FastDFS: fastdfs.connection_pool_max_total=" + this.fdfsProp.getConnectionPoolMaxTotal());
    this.logger.info("FastDFS: fastdfs.connection_pool_max_idle=" + this.fdfsProp.getConnectionPoolMaxIdle());
    this.logger.info("FastDFS: fastdfs.connection_pool_min_idle=" + this.fdfsProp.getConnectionPoolMinIdle());
    this.logger.info("FastDFS: fastdfs.nginx_servers=" + this.fdfsProp.getNginxServers());

    this.prop.put("fastdfs.connect_timeout_in_seconds", this.fdfsProp.getConnectTimeout());
    this.prop.put("fastdfs.network_timeout_in_seconds", this.fdfsProp.getNetworkTimeout());
    this.prop.put("fastdfs.charset", this.fdfsProp.getCharset());
    this.prop.put("fastdfs.http_anti_steal_token", this.fdfsProp.getHttpAntiStealToken());
    this.prop.put("fastdfs.http_secret_key", this.fdfsProp.getHttpSecretKey());
    this.prop.put("fastdfs.http_tracker_http_port", this.fdfsProp.getHttpTrackerHttpPort());
    this.prop.put("fastdfs.tracker_servers", this.fdfsProp.getTrackerServers());
    ClientGlobal.initByProperties(this.prop);
  }
  /**
   *  Display initialization information 
   */
  private void info() {
    this.logger.info("FastDFS parameter: ConnectionPoolMaxTotal ==> " + this.pool.getMaxTotal());
    this.logger.info("FastDFS parameter: ConnectionPoolMaxIdle ==> " + this.pool.getMaxIdle());
    this.logger.info("FastDFS parameter: ConnectionPoolMinIdle ==> " + this.pool.getMinIdle());
    this.logger.info("FastDFS parameter: NginxServer ==> " + Arrays.toString(this.nginxServers));
    this.logger.info(ClientGlobal.configInfo());
  }

  /**
   *  Create a connection pool 
   */
  private void create() {
    this.config = new GenericObjectPoolConfig();
    this.logger.info("FastDFS Client: Creating connection pool...");
    this.config.setMaxTotal(Integer.parseInt(this.fdfsProp.getConnectionPoolMaxTotal()));
    this.config.setMaxIdle(Integer.parseInt(this.fdfsProp.getConnectionPoolMaxIdle()));
    this.config.setMinIdle(Integer.parseInt(this.fdfsProp.getConnectionPoolMinIdle()));
    StorageClientFactory factory = new StorageClientFactory();
    this.pool = new GenericObjectPool<StorageClient>(factory, this.config);
    this.nginxServers = this.fdfsProp.getNginxServers().split(",");
  }

  /**
   * Nginx Server load balancing algorithm 
   *
   * @param servers  Server address 
   * @param address  Client IP Address 
   * @return  Available server addresses 
   */
  private String getNginxServer(String[] servers, String address) {
    int size = servers.length;
    int i = address.hashCode();
    int index = abs(i % size);
    return servers[index];
  }

  /**
   *  Download with anti-theft chain 
   *
   * @param fileGroup     Filegroup name 
   * @param remoteFileName  Remote file name 
   * @param clientIpAddress  Client IP Address 
   * @return  Complete URL Address 
   */
  public String autoDownloadWithToken(String fileGroup, String remoteFileName, String clientIpAddress) throws Exception {
    int ts = (int) (System.currentTimeMillis() / 1000);
    String token = ProtoCommon.getToken(remoteFileName, ts, ClientGlobal.getG_secret_key());
    String nginx = this.getNginxServer(this.nginxServers, clientIpAddress);
    return "http://" + nginx + "/" + fileGroup + "/" + remoteFileName + "?token=" + token + "&ts=" + ts;
  }

  /**
   *  Upload files, suitable for uploading pictures 
   *
   * @param buffer  Byte array 
   * @param ext   Extensions 
   * @return  Filegroup name and ID
   */
  public String[] autoUpload(byte[] buffer, String ext) throws Exception {
    String[] upload = this.upload(buffer, ext, null);
    return upload;
  }

  /**
   *  Download without anti-theft chain. If the anti-theft chain is turned on, this method will throw an exception 
   *
   * @param fileGroup     Filegroup name 
   * @param remoteFileName  Remote file ID
   * @param clientIpAddress  Client IP Address, according to the client IP To distribute Nginx Server 
   * @return  Complete URL Address 
   */
  public String autoDownloadWithoutToken(String fileGroup, String remoteFileName, String clientIpAddress) throws Exception {
    if (ClientGlobal.getG_anti_steal_token()) {
      this.logger.error("FastDFS Client: You've turned on Token authentication.");
      throw new Exception("You've turned on Token authentication.");
    }
    String nginx = this.getNginxServer(this.nginxServers, clientIpAddress);
    return "http://" + nginx + fileGroup + "/" + remoteFileName;
  }

  //  There are still many ways to go, so don't 11 Shows the 
}

To use the convenient configuration hints in IDEA, we need to create a metadata file (resources/spring-configuration-metadata. json):

{
 "groups": [
  {
   "name": "fastdfs",
   "type": "com.bluemiaomiao.properties.FastdfsProperties",
   "sourceType": "com.bluemiaomiao.properties.FastdfsProperties"
  }
 ],
 "properties": [
  {
   "name": "connectTimeout",
   "type": "java.lang.String",
   "sourceType": "com.bluemiaomiao.properties.FastdfsProperties",
   "defaultValue": "5"
  },
  {
   "name": "networkTimeout",
   "type": "java.lang.String",
   "sourceType": "com.bluemiaomiao.properties.FastdfsProperties",
   "defaultValue": "30"
  },
  {
   "name": "charset",
   "type": "java.lang.String",
   "defaultValue": "UTF-8"
  },
  {
   "name": "httpAntiStealToken",
   "type": "java.lang.String",
   "sourceType": "com.bluemiaomiao.properties.FastdfsProperties",
   "defaultValue": "false"
  },
  {
   "name": "httpSecretKey",
   "type": "java.lang.String",
   "sourceType": "com.bluemiaomiao.properties.FastdfsProperties"
  },
  {
   "name": "httpTrackerHttpPort",
   "type": "java.lang.Integer",
   "sourceType": "com.bluemiaomiao.properties.FastdfsProperties"
  },
  {
   "name": "trackerServers",
   "type": "java.lang.String",
   "sourceType": "com.bluemiaomiao.properties.FastdfsProperties"
  },
  {
   "name": "connectionPoolMaxTotal",
   "type": "java.lang.Integer",
   "sourceType": "com.bluemiaomiao.properties.FastdfsProperties",
   "defaultValue": "18"
  },
  {
   "name": "connectionPoolMaxIdle",
   "type": "java.lang.Integer",
   "sourceType": "com.bluemiaomiao.properties.FastdfsProperties",
   "defaultValue": "18"
  },
  {
   "name": "connectionPoolMinIdle",
   "type": "java.lang.Integer",
   "sourceType": "com.bluemiaomiao.properties.FastdfsProperties",
   "defaultValue": "2"
  },
  {
   "name": "nginxServers",
   "type": "java.lang.String",
   "sourceType": "com.bluemiaomiao.properties.FastdfsProperties"
  }
 ],
 "hints": [
  {
   "name": "http_anti_steal_token",
   "values": [
    {
     "value": "false"
    },
    {
     "value": "true"
    }
   ]
  }
 ]
}

Apply to the project

Create an SpringBoot project, check the Web option, and select version 1.5. 20

Go to the project directory of the scenario initiator and execute mvn clean install to install it locally

Add dependencies to the POM. xml file:

<dependency>
  <groupId>com.bluemiaomiao</groupId>
  <artifactId>fastdfs-spring-boot-starter</artifactId>
  <version>1.0-SNAPSHOT</version>
</dependency>

Remember to turn on the automatic import function of IDEA

Create configuration file application. properties

fastdfs.nginx-servers=192.168.80.2:8000,192.168.80.3:8000,192.168.80.4:8000
fastdfs.tracker-servers=192.168.80.2:22122,192.168.80.3:22122,192.168.80.4:22122
fastdfs.http-secret-key=2scPwMPctXhbLVOYB0jyuyQzytOofmFCBIYe65n56PPYVWrntxzLIDbPdvDDLJM8QHhKxSGWTcr+9VdG3yptkw
fastdfs.http-anti-steal-token=true
fastdfs.http-tracker-http-port=8080
fastdfs.network-timeout=30
fastdfs.connect-timeout=5
fastdfs.connection-pool-max-idle=18
fastdfs.connection-pool-min-idle=2
fastdfs.connection-pool-max-total=18
fastdfs.charset=UTF-8

Or use application. yml

fastdfs:
 charset: UTF-8
 connect-timeout: 5
 http-secret-key: 2scPwMPctXhbLVOYB0jyuyQzytOofmFCBIYe65n56PPYVWrntxzLIDbPdvDDLJM8QHhKxSGWTcr+9VdG3yptkw
 network-timeout: 30
 http-anti-steal-token: true
 http-tracker-http-port: 8080
 connection-pool-max-idle: 20
 connection-pool-max-total: 20
 connection-pool-min-idle: 2
 nginx-servers: 192.168.80.2:8000,192.168.80.3:8000,192.168.80.4:8000
 tracker-servers: 192.168.80.2:22122,192.168.80.3:22122,192.168.80.4:22122

Creating a Controller Class Test Method

// EnableFastdfsClient.java
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Import(FastdfsAutoConfiguration.class)
@Documented
public @interface EnableFastdfsClient {
}

0

Project home page: https://github.com/bluemiaomiao/fastdfs-spring-boot-starter

Home page of domestic projects: https://gitee.com/bluemiaomiao/fastdfs-spring-boot-starter