Summary of FREQUENTLY asked questions based on Elasticsearch5.4

Recently, es1EN5.4 (ES) was used in the project, which is a relatively new version. A lot of problems occurred in the process of using es1EN5.4. It was a headache, but the problem was finally solved.

Question 1: ESClient is slow to acquire and cannot acquire Client: failed to create a child event loop

Due to the needs of the business didn't upload all files will be 1 times ES index, every index + 1 times to obtain a connection and operation, especially when large quantities of, access to the number of times the obviously very much, and the problems of the operation of the main reason is that we are in the loop frequently ES, such as 1, 100 a batch file, we're going to get 100 times, in order to reduce ES Client acquisition time, eventually adopted a scheme, that is at the time of service startup initialization connection, once get, and then directly behind calls, the whole batch file upload is completed, Finally add ES index instead of adding file by file. This approach obviously eliminates the need to acquire connections for each batch, greatly improving execution efficiency.

First, we initialize static ES Client in the startup class when the service starts:

private static ElasticSearchUtil ElasticSearchUtil=new ElasticSearchUtil();
public static TransportClient client=ElasticSearchUtil.getClient();

Then call it directly when it's in use:

Client client=Main.client;

This can significantly reduce the number of connections to ES Client, thus improving efficiency.

ES code is as follows:

public TransportClient getClient() {
String[] ipArr = configUtil.getValue("ESIP").split(",");
Settings settings = Settings.builder().put("thread_pool.generic.core",5)
　　　　.put("thread_pool.generic.max", 10)
　　　　.put("processors", 5)
　　　　.put(Constants.ESCLUSTERNAME,configUtil.getValue("clusterName")).build();

TransportClient client = new PreBuiltTransportClient(settings);
for (String ip : ipArr) {
TransportAddress address = new InetSocketTransportAddress
　　(InetAddresses.forString(ip),9300);
client.addTransportAddresses(address);
}
　　return client;
}

Question 2: out of memory: java. lang. OutOfMemory: unable to create new native thread

Occurred in the process of project development, memory is a troubling one thing at a time, and in the process of using ES, met, and very often, especially when large quantities of pressure test are not bottom go to, from the aspects of jvm memory tuning to a number of ways, there is no effect, the problem is still not solved, finally looking at the source, found a reason, and an error and it is with ES in use process, automatically creating a large number of threads, beyond the accomodation of the system, so led to a memory leak, the source code to find: The number of threads created by ES can be controlled by setting. Here is the default number of threads created by ES:

thread_pool.generic.core= The default value ---4
thread_pool.generic.max= The default value --
min(512,max(4*processor The number, 128))
processor The number =CPU the processor The number 

Our CPU is 10 core 40 threads

According to the calculation results, if the default value is used, the number of threads ES can create is a large number, which is far beyond the capacity of the system itself. The main reason is to adjust the value of setting. After adjustment, the default value of ES663en is changed as follows:

Settings settings = Settings.builder().put("thread_pool.generic.core",5)
.put("thread_pool.generic.max", 10)
.put("processors", 5)  .put(Constants.ESCLUSTERNAME,configUtil.getValue("clusterName")).build();
 This is the previous one 
Settings settings = Settings.builder().put("thread_pool.generic.core",5)
.put(Constants.ESCLUSTERNAME,configUtil.getValue("clusterName")).build();

In our tests, ES created a very small number of threads and met our development requirements, and there were no more out-of-memory issues.