spring boot encapsulates HttpClient's sample code
- 2020-12-22 17:37:34
- OfStack
Recently used by HttpClient, see the official documentation under 1: HttpClient implementations are expected to be thread safe. It is recommended that the same instance of this class is reused for multiple request executions, translation means: The implementation of HttpClient is thread-safe and can reuse the same instance to execute multiple requests. When confronted with this description, we should think of the need to encapsulate HttpClient. As spring boot is used, it is combined with spring boot to encapsulate HttpClient.
1. Request retry handler(request retry processing)
In order for the custom exception mechanism to take effect, the HttpRequestRetryHandler interface needs to be implemented as follows:
import java.io.IOException;
import java.io.InterruptedIOException;
import java.net.UnknownHostException;
import javax.net.ssl.SSLException;
import javax.net.ssl.SSLHandshakeException;
import org.apache.http.HttpEntityEnclosingRequest;
import org.apache.http.HttpRequest;
import org.apache.http.NoHttpResponseException;
import org.apache.http.client.HttpRequestRetryHandler;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.conn.ConnectTimeoutException;
import org.apache.http.protocol.HttpContext;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
/**
* Description: HttpClient Retry processing mechanism
*/
@Configuration
public class MyhttpRequestRetryHandler {
@Value("${httpclient.config.retryTime}")// It is recommended that @ConfigurationProperties(prefix="httpclient.config") Way, easy to reuse
private int retryTime;
@Bean
public HttpRequestRetryHandler httpRequestRetryHandler() {
// Request retry
final int retryTime = this.retryTime;
return new HttpRequestRetryHandler() {
public boolean retryRequest(IOException exception, int executionCount, HttpContext context) {
// Do not retry if over max retry count, If the number of retries exceeds retryTime, The request is not retried
if (executionCount >= retryTime) {
return false;
}
// The server broke the client connection exception
if (exception instanceof NoHttpResponseException) {
return true;
}
// time out Timeout retry
if (exception instanceof InterruptedIOException) {
return true;
}
// Unknown host
if (exception instanceof UnknownHostException) {
return false;
}
// Connection refused
if (exception instanceof ConnectTimeoutException) {
return false;
}
// SSL handshake exception
if (exception instanceof SSLException) {
return false;
}
HttpClientContext clientContext = HttpClientContext.adapt(context);
HttpRequest request = clientContext.getRequest();
if (!(request instanceof HttpEntityEnclosingRequest)) {
return true;
}
return false;
}
};
}
}
2. Pooling connection manager(Connection pool Management)
PoolingHttpClientConnectionManager is used to manage connection pools on the client side and can service requests from multiple threads as follows:
import org.apache.http.config.Registry;
import org.apache.http.config.RegistryBuilder;
import org.apache.http.conn.socket.ConnectionSocketFactory;
import org.apache.http.conn.socket.LayeredConnectionSocketFactory;
import org.apache.http.conn.socket.PlainConnectionSocketFactory;
import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
public class MyPoolingHttpClientConnectionManager {
/**
* Maximum number of connections in the connection pool
*/
@Value("${httpclient.config.connMaxTotal}")
private int connMaxTotal = 20;
/**
*
*/
@Value("${httpclient.config.maxPerRoute}")
private int maxPerRoute = 20;
/**
* Connection survival time, in units of s
*/
@Value("${httpclient.config.timeToLive}")
private int timeToLive = 60;
@Bean
public PoolingHttpClientConnectionManager poolingClientConnectionManager(){
PoolingHttpClientConnectionManager poolHttpcConnManager = new PoolingHttpClientConnectionManager(60, TimeUnit.SECONDS);
// Maximum number of connections
poolHttpcConnManager.setMaxTotal(this.connMaxTotal);
// Routing base
poolHttpcConnManager.setDefaultMaxPerRoute(this.maxPerRoute);
return poolHttpcConnManager;
}
}
Note: When an HttpClient instance is no longer needed and is about to go out of scope, it is important to close its connection manager to ensure that all the connections that the manager keeps active are closed and to free up system resources allocated by those connections
The constructor for the PoolingHttpClientConnectionManager class above is as follows:
public PoolingHttpClientConnectionManager(final long timeToLive, final TimeUnit tunit) {
this(getDefaultRegistry(), null, null ,null, timeToLive, tunit);
}
private static Registry<ConnectionSocketFactory> getDefaultRegistry() {
return RegistryBuilder.<ConnectionSocketFactory>create()
.register("http", PlainConnectionSocketFactory.getSocketFactory())
.register("https", SSLConnectionSocketFactory.getSocketFactory())
.build();
}
There are two maximum number of connections in the PoolingHttpClientConnectionManager configuration, controlling the total and the maximum number of connections per route, respectively. Without explicit Settings, the default route is to allow a maximum of two connection per route, with the total number of connection not exceeding 20. This value is not sufficient for many concurrent applications, and the appropriate value must be set according to the actual situation. The idea is similar to how the thread pool size is set. If all connection requests are to the same url, then the value of MaxPerRoute can be set to MaxTotal1, so that connections can be reused more efficiently
Special note: In order to reuse one connection, the system resources occupied by it must be released correctly, and the release method is as follows:
If you are using outputStream, make sure that the entire entity is used by write out, and if it is inputStream, remember to call inputStream.close () at the end. Or use EntityUtils.consume (entity) or EntityUtils.consumeQuietly (entity) to let entity be completely exhausted (the latter does not throw exceptions) to do this 1 job. It is also convenient to have an toString method in EntityUtils (calling this method will eventually drop inputStream close automatically, but in the actual test, this will cause the connection not to be released), but only if you can be sure that the received entity is not particularly large. Without the entire entity being used by fully consumed, the connection would not be reusable and would soon be blocked because no available connection timeout was taken from the connection pool (since the state of the connection would be 1 straight from leased, the state in use). So if you want to reuse connection, 1 must 1 must remember to drop entity fully consume, as long as the eof of stream is detected, it will automatically call the releaseConnection method of ConnectionHolder
3. Connection keep alive strategy(Stay connected strategy)
The HTTP specification does not specify how long persistent connections may and should remain alive. Some HTTP servers use non-standard Keep-ES123en headers to communicate to the client for the length of time (in seconds) that they intend to remain connected to the server. HttpClient can use this information. If the ES125en-ES126en header is not present in the response, HttpClient assumes that the connection can remain active indefinitely. However, many HTTP servers used in general are configured to remove persistent connections after a period of inactivity to save system resources without notifying the client. If the default policy is too optimistic, you may need to provide a custom keep-alive policy, as follows:
import org.apache.http.HeaderElement;
import org.apache.http.HeaderElementIterator;
import org.apache.http.HttpResponse;
import org.apache.http.conn.ConnectionKeepAliveStrategy;
import org.apache.http.message.BasicHeaderElementIterator;
import org.apache.http.protocol.HTTP;
import org.apache.http.protocol.HttpContext;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
/**
* Description: Connection retention policy
* @author chhliu
*/
@Configuration
public class MyconnectionKeepAliveStrategy {
@Value("${httpclient.config.keepAliveTime}")
private int keepAliveTime = 30;
@Bean("connectionKeepAliveStrategy")
public ConnectionKeepAliveStrategy connectionKeepAliveStrategy() {
return new ConnectionKeepAliveStrategy() {
public long getKeepAliveDuration(HttpResponse response, HttpContext context) {
// Honor 'keep-alive' header
HeaderElementIterator it = new BasicHeaderElementIterator(
response.headerIterator(HTTP.CONN_KEEP_ALIVE));
while (it.hasNext()) {
HeaderElement he = it.nextElement();
String param = he.getName();
String value = he.getValue();
if (value != null && param.equalsIgnoreCase("timeout")) {
try {
return Long.parseLong(value) * 1000;
} catch (NumberFormatException ignore) {
}
}
}
return 30 * 1000;
}
};
}
}
Note: long connection is not used in all cases, especially now system, are mostly deployed in multiple servers, and it has the function of load balancing, if we are on a visit, 1 straight maintain a long connection, 1 hajek said server hang up and will affect the client, at the same time also can't fully use of server load balancing feature, rather short connection more favorable 1, these need to be decided according to the specific needs, instead of one word.
4. HttpClient proxy configuration(Agent configuration)
To configure the agent, the code is as follows:
import org.apache.http.HttpHost;
import org.apache.http.impl.conn.DefaultProxyRoutePlanner;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
/**
* Description: HttpClient The agent
* @author chhliu
*/
@Configuration
public class MyDefaultProxyRoutePlanner {
// The agent's host address
@Value("${httpclient.config.proxyhost}")
private String proxyHost;
// The port number of the agent
@Value("${httpclient.config.proxyPort}")
private int proxyPort = 8080;
@Bean
public DefaultProxyRoutePlanner defaultProxyRoutePlanner(){
HttpHost proxy = new HttpHost(this.proxyHost, this.proxyPort);
return new DefaultProxyRoutePlanner(proxy);
}
}
HttpClient not only supports simple direct connections, complex routing policies, and proxies. HttpRoutePlanner is based on the http context, the client to server routing calculation strategy, generally no proxy, you do not need to set this thing. Here is a key concept - Route: in HttpClient, an Route refers to a running environment machine -
>
1 circuit of target machine host, i.e. if target url's host is the same, then their route is the same
5. RequestConfig
The various configurations used to set up the request are as follows:
import org.apache.http.client.config.RequestConfig;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
public class MyRequestConfig {
@Value("${httpclient.config.connectTimeout}")
private int connectTimeout = 2000;
@Value("${httpclient.config.connectRequestTimeout}")
private int connectRequestTimeout = 2000;
@Value("${httpclient.config.socketTimeout}")
private int socketTimeout = 2000;
@Bean
public RequestConfig config(){
return RequestConfig.custom()
.setConnectionRequestTimeout(this.connectRequestTimeout)
.setConnectTimeout(this.connectTimeout)
.setSocketTimeout(this.socketTimeout)
.build();
}
}
RequestConfig is some configuration of request. The most important of these are three timeout periods, all of which are 0 by default (if you do not set request's Config, you will use HttpClientParamConfig's getRequestConfig during execute with the default parameter), which means an infinite wait and can easily cause all requests to block in this place and wait indefinitely. The three timeout periods are:
a, connectionRequestTimeout - Timeout to fetch a connection from the connection pool
This time is defined as the timeout to take a connection out of the connection pool managed by ConnectionManager. If there are no connections available in the connection pool, request is blocked, waits for the maximum amount of time for connectionRequestTimeout, and if not already served, throws an ConnectionPoolTimeoutException exception and does not continue to wait.
b, connectTimeout - Connection timeout
This time defines the timeout for establishing a connection to the server over the network, that is, the wait time for a connection to the connected target url after a connection is made from the connection pool. A timeout occurs and an ConnectionTimeoutException exception is thrown.
c, socketTimeout - Request timeout time
This time defines the timeout for socket read data, that is, the wait time between connecting to the server and getting the response data from the server, or the return wait time between connecting to 1 url and getting response. A timeout occurs and an SocketTimeoutException exception is thrown.
6. Instantiate HttpClient
HttpClient is instantiated by implementing FactoryBean, the code is as follows:
import org.apache.http.client.HttpRequestRetryHandler;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.conn.ConnectionKeepAliveStrategy;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.impl.conn.DefaultProxyRoutePlanner;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
import org.springframework.beans.factory.DisposableBean;
import org.springframework.beans.factory.FactoryBean;
import org.springframework.beans.factory.InitializingBean;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
/**
* Description: HttpClient Client encapsulation
*/
@Service("httpClientManagerFactoryBen")
public class HttpClientManagerFactoryBen implements FactoryBean<CloseableHttpClient>, InitializingBean, DisposableBean {
/**
* FactoryBean The generated target object
*/
private CloseableHttpClient client;
@Autowired
private ConnectionKeepAliveStrategy connectionKeepAliveStrategy;
@Autowired
private HttpRequestRetryHandler httpRequestRetryHandler;
@Autowired
private DefaultProxyRoutePlanner proxyRoutePlanner;
@Autowired
private PoolingHttpClientConnectionManager poolHttpcConnManager;
@Autowired
private RequestConfig config;
// When the context is destroyed, destroy HttpClient The instance
@Override
public void destroy() throws Exception {
/*
* call httpClient.close() Will first shut down connection manager And then release that HttpClient All the resources that are occupied,
* Close all in use or idle connection Including the underlying socket . Because here's what it's used for connection manager Shut down,
* So we'll do it next time http When you ask, do it again new1 a connection manager to build1 a HttpClient,
* That's when you need to close and create Client In the case of, connection manager It can't be a singleton .
*/
if(null != this.client){
this.client.close();
}
}
@Override// Initializing instance
public void afterPropertiesSet() throws Exception {
/*
* Recommended use here HttpClients.custom Ways to create HttpClientBuilder "Instead of using it HttpClientBuilder.create() Method to create HttpClientBuilder
* From the official documents, HttpClientBuilder Non-thread-safe, but HttpClients indeed Immutable The, immutable Not only does an object ensure that its state does not change,
* It can also be shared by other threads without using the locking mechanism
*/
this.client = HttpClients.custom().setConnectionManager(poolHttpcConnManager)
.setRetryHandler(httpRequestRetryHandler)
.setKeepAliveStrategy(connectionKeepAliveStrategy)
.setRoutePlanner(proxyRoutePlanner)
.setDefaultRequestConfig(config)
.build();
}
// Returns the type of the instance
@Override
public CloseableHttpClient getObject() throws Exception {
return this.client;
}
@Override
public Class<?> getObjectType() {
return (this.client == null ? CloseableHttpClient.class : this.client.getClass());
}
// The built instance is a singleton
@Override
public boolean isSingleton() {
return true;
}
}
7. Add configuration files
# The agent's host
httpclient.config.proxyhost=xxx.xx.xx.xx
# The proxy port
httpclient.config.proxyPort=8080
# Number of connection timeouts or abnormal retries
httpclient.config.retryTime=3
# Long connection holding time, in units of s
httpclient.config.keepAliveTime=30
# Maximum number of connections in the connection pool
httpclient.config.connMaxTotal=20
httpclient.config.maxPerRoute=20
# Connection timeout, unit ms
httpclient.config.connectTimeout=2000
# Request timeout
httpclient.config.connectRequestTimeout=2000
# sock timeout
httpclient.config.socketTimeout=2000
# Connection survival time, unit s
httpclient.config.timeToLive=60
Test 8.
The test code is as follows:
import java.io.IOException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import javax.annotation.Resource;
import org.apache.http.Consts;
import org.apache.http.ParseException;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.util.EntityUtils;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;
@RunWith(SpringRunner.class)
@SpringBootTest
public class HttpClientManagerFactoryBenTest {
// injection HttpClient The instance
@Resource(name = "httpClientManagerFactoryBen")
private CloseableHttpClient client;
@Test
public void test() throws ClientProtocolException, IOException, InterruptedException{
ExecutorService service = Executors.newFixedThreadPool(2);
for(int i=0; i<10; i++){
service.submit(new Runnable() {
@Override
public void run() {
System.out.println("the current thread is:"+Thread.currentThread().getName());
HttpEntity entity = null;
try {
HttpGet get = new HttpGet("https://localhost:8080/testjson");
// through httpclient the execute submit request And use CloseableHttpResponse Accept the return message
CloseableHttpResponse response = client.execute(get);
System.out.println("client object:"+client);
entity = response.getEntity();
System.out.println("============"+EntityUtils.toString(entity, Consts.UTF_8)+"=============");
EntityUtils.consumeQuietly(entity);// Release the connection
} catch (ClientProtocolException e) {
e.printStackTrace();
} catch (ParseException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally{
if(null != entity){// Release the connection
EntityUtils.consumeQuietly(entity);
}
}
}
});
}
Thread.sleep(60000);
}
}
With the above steps, HttpClient is basically encapsulated. If you need more detail, you can follow the above idea and gradually perfect HttpClient into HttpClientTemplate, because CloseableHttpClient uses callback mechanism internally, similar to JdbcTemplate or RedisTemplate, until it is available to provide services as spring boot starter.