Some Suggestions for zookeeper Service Optimization

1. Snapshot files and transaction log files are hung on different disks. In the zoo. cfg file, dataDir holds the snapshot data and dataLogDir holds the transaction log. zookeeper update operation process: First write transaction log, then write memory, periodically fall to disk (refresh memory to snapshot file). Transaction logs have a great impact on the performance of write requests, ensuring that the disk where dataLogDir is located has good performance and no competitors.

2. The default jvm is not configured with Xmx, Xms and other information. You can create java. env files in the conf directory (memory heap space 1 must be smaller than machine memory, avoid using swap)

export JVMFLAGS="-Xms2048m -Xmx2048m $JVMFLAGS"

3. Make zookeeper log by day to avoid zookeeper. out file being too large.

zkEnv.sh File log output mode from CONSOLE Replace with ROLLINGFILE ;

if [ "x${ZOO_LOG4J_PROP}" = "x" ]
then
#  ZOO_LOG4J_PROP="INFO,CONSOLE"
  ZOO_LOG4J_PROP="INFO,ROLLINGFILE"
fi

conf/log4j. properties set to generate files on a daily basis DailyRollingFileAppender

#zookeeper.root.logger=INFO, CONSOLE
zookeeper.root.logger=INFO, ROLLINGFIL

log4j.appender.ROLLINGFILE=org.apache.log4j.DailyRollingFileAppender
log4j.appender.ROLLINGFILE.Threshold=${zookeeper.log.threshold}
log4j.appender.ROLLINGFILE.File=${zookeeper.log.dir}/${zookeeper.log.file}
log4j.appender.ROLLINGFILE.DatePattern='.'yyyy-MM-dd
# Max log file size of 10MB
#log4j.appender.ROLLINGFILE.MaxFileSize=10MB

4. In the zoo. cfg file skipACL=yes Ignoring ACL authentication can reduce the related operations of permission authentication and improve performance by 1 point.

5. In the zoo. cfg file forceSync=no This is very helpful to improve the performance of write requests, which means that the data of each write request should be solidified from pagecache to disk before it can be regarded as a successful write return. When the number of write requests reaches a certain level, the subsequent write requests will wait for the forceSync operation of the previous write request, resulting in a certain delay. If a low latency write request is pursued, configure forceSync=no, and the data is returned after writing to pagecache. However, when the machine is powered off, the data in pagecache may be lost.

The default is forceSync=yes, and yes can be set fsync.warningthresholdms=50 If fsync exceeds 50ms, an warn log will be output in zookeeper. out (forceSync = yes valid).

6. globalOutstandingLimit=100000 Too many client connections, limit client requests and avoid OOM

7. In the zoo. cfg file preAllocSize=64M Pre-allocated size of log file; snapCount=100,000 How many times to write transactions, generate 1 snapshot. If snapshots are generated frequently, this parameter can be adjusted appropriately.

The application of zk advocates reading more than writing, has better performance (10: 1), and stores metadata to coordinate distributed data. It is better to write too frequently and use cache

8. Automatic cleanup of log files (manual cleanup if performance is desired)

autopurge.snapRetainCount=3 # The number of snapshots to retain in dataDir
autopurge.purgeInterval=24 # Purge task interval in hours Set to "0" to disable auto purge feature

Summarize