Zabbix partition optimization of Mysql optimization
- 2021-09-20 21:44:29
- OfStack
The biggest bottleneck of using zabbix lies in the database. If the data storage and alarm of zabbix are well maintained, zabbix can be well used to build a monitoring system. At present, the data of zabbix is mainly stored in two tables of history and trends. As time goes by, these two tables become very large, and their performance will be very poor, which will affect the use of monitoring. Tuning MySQL can greatly improve the performance of Zabbix. This paper uses the method of partitioning MySQL for tuning.
Principle
Partition tables such as history and trends in zabbix, partition by date, one per day, and reserve partitions for 90 days.
Detailed operation steps
Operation impact: Online operation, MySQL read and write slow, Zabbix performance slow, impact time according to the small data changes, 1 generally in 2 hours or so.
Step 1
Log in to the database of zabbix server, and unify the configuration of MySQL
cat > /etc/my.cnf<<EOF
[mysqld]
datadir=/data/mysql
socket=/var/lib/mysql/mysql.sock
default-storage-engine = innodb
collation-server = utf8_general_ci
init-connect = 'SET NAMES utf8'
character-set-server = utf8
symbolic-links=0
max_connections=4096
innodb_buffer_pool_size=12G
max_allowed_packet = 32M
join_buffer_size=2M
sort_buffer_size=2M
query_cache_size = 64M
query_cache_limit = 4M
thread_concurrency = 8
table_open_cache=1024
innodb_flush_log_at_trx_commit = 0
long_query_time = 1
log-slow-queries =/data/mysql/mysql-slow.log
[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid
#[mysql]
#socket=/data/mysql/mysql.sock
#
# include all files from the config directory
#
!includedir /etc/my.cnf.d
EOFNote: 1 Be sure to modify innodb_buffer_pool_size= 1/3 of physical memory
Step 2
Confirm the version of zabbix first. Version 1 of zabbix must be greater than 3.2. 0 in this operation. This operation cannot be installed for versions less than 3.2, and the online default is zabbix-3. 2.6.
a, Import Stored Procedure
#cat partition.sql
DELIMITER $$
CREATE PROCEDURE `partition_create`(SCHEMANAMEvarchar(64), TABLENAME varchar(64), PARTITIONNAME varchar(64), CLOCK int)
BEGIN
/*
SCHEMANAME = The DB schema in which to make changes
TABLENAME = The table with partitions to potentially delete
PARTITIONNAME = The name of the partition to create
*/
/*
Verify that the partition does not already exist
*/
DECLARE RETROWS INT;
SELECT COUNT(1) INTO RETROWS
FROM information_schema.partitions
WHERE table_schema = SCHEMANAME AND table_name = TABLENAME ANDpartition_description >= CLOCK;
IF RETROWS = 0 THEN
/*
1. Print a messageindicating that a partition was created.
2. Create the SQL to createthe partition.
3. Execute the SQL from #2.
*/
SELECT CONCAT( "partition_create(", SCHEMANAME, ",",TABLENAME, ",", PARTITIONNAME, ",", CLOCK, ")" )AS msg;
SET @sql = CONCAT( 'ALTER TABLE ', SCHEMANAME, '.', TABLENAME, ' ADDPARTITION (PARTITION ', PARTITIONNAME, ' VALUES LESS THAN (', CLOCK, '));' );
PREPARE STMT FROM @sql;
EXECUTE STMT;
DEALLOCATE PREPARE STMT;
END IF;
END$$
DELIMITER ;
DELIMITER $$
CREATE PROCEDURE `partition_drop`(SCHEMANAMEVARCHAR(64), TABLENAME VARCHAR(64), DELETE_BELOW_PARTITION_DATE BIGINT)
BEGIN
/*
SCHEMANAME = The DB schema in which tomake changes
TABLENAME = The table with partitions to potentially delete
DELETE_BELOW_PARTITION_DATE = Delete any partitions with names that aredates older than this one (yyyy-mm-dd)
*/
DECLARE done INT DEFAULT FALSE;
DECLARE drop_part_name VARCHAR(16);
/*
Get a list of all the partitions that are older than the date
in DELETE_BELOW_PARTITION_DATE. All partitions are prefixed with
a "p", so use SUBSTRING TOget rid of that character.
*/
DECLARE myCursor CURSOR FOR
SELECT partition_name
FROM information_schema.partitions
WHERE table_schema = SCHEMANAME AND table_name = TABLENAME ANDCAST(SUBSTRING(partition_name FROM 2) AS UNSIGNED) <DELETE_BELOW_PARTITION_DATE;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = TRUE;
/*
Create the basics for when we need to drop the partition. Also, create
@drop_partitions to hold a comma-delimited list of all partitions that
should be deleted.
*/
SET @alter_header = CONCAT("ALTER TABLE ", SCHEMANAME,".", TABLENAME, " DROP PARTITION ");
SET @drop_partitions = "";
/*
Start looping through all the partitions that are too old.
*/
OPEN myCursor;
read_loop: LOOP
FETCH myCursor INTO drop_part_name;
IF done THEN
LEAVE read_loop;
END IF;
SET @drop_partitions = IF(@drop_partitions = "",drop_part_name, CONCAT(@drop_partitions, ",", drop_part_name));
END LOOP;
IF @drop_partitions != "" THEN
/*
1. Build the SQL to drop allthe necessary partitions.
2. Run the SQL to drop thepartitions.
3. Print out the tablepartitions that were deleted.
*/
SET @full_sql = CONCAT(@alter_header, @drop_partitions, ";");
PREPARE STMT FROM @full_sql;
EXECUTE STMT;
DEALLOCATE PREPARE STMT;
SELECT CONCAT(SCHEMANAME, ".", TABLENAME) AS `table`,@drop_partitions AS `partitions_deleted`;
ELSE
/*
No partitions are beingdeleted, so print out "N/A" (Not applicable) to indicate
that no changes were made.
*/
SELECT CONCAT(SCHEMANAME, ".", TABLENAME) AS `table`,"N/A" AS `partitions_deleted`;
END IF;
END$$
DELIMITER ;
DELIMITER $$
CREATE PROCEDURE`partition_maintenance`(SCHEMA_NAME VARCHAR(32), TABLE_NAME VARCHAR(32),KEEP_DATA_DAYS INT, HOURLY_INTERVAL INT, CREATE_NEXT_INTERVALS INT)
BEGIN
DECLARE OLDER_THAN_PARTITION_DATE VARCHAR(16);
DECLARE PARTITION_NAME VARCHAR(16);
DECLARE OLD_PARTITION_NAME VARCHAR(16);
DECLARE LESS_THAN_TIMESTAMP INT;
DECLARE CUR_TIME INT;
CALL partition_verify(SCHEMA_NAME,TABLE_NAME, HOURLY_INTERVAL);
SET CUR_TIME = UNIX_TIMESTAMP(DATE_FORMAT(NOW(), '%Y-%m-%d 00:00:00'));
SET @__interval = 1;
create_loop: LOOP
IF @__interval > CREATE_NEXT_INTERVALS THEN
LEAVE create_loop;
END IF;
SET LESS_THAN_TIMESTAMP = CUR_TIME + (HOURLY_INTERVAL * @__interval *3600);
SET PARTITION_NAME = FROM_UNIXTIME(CUR_TIME + HOURLY_INTERVAL *(@__interval - 1) * 3600, 'p%Y%m%d%H00');
IF(PARTITION_NAME != OLD_PARTITION_NAME) THEN
CALLpartition_create(SCHEMA_NAME, TABLE_NAME, PARTITION_NAME, LESS_THAN_TIMESTAMP);
END IF;
SET @__interval=@__interval+1;
SET OLD_PARTITION_NAME = PARTITION_NAME;
END LOOP;
SET OLDER_THAN_PARTITION_DATE=DATE_FORMAT(DATE_SUB(NOW(), INTERVALKEEP_DATA_DAYS DAY), '%Y%m%d0000');
CALL partition_drop(SCHEMA_NAME, TABLE_NAME, OLDER_THAN_PARTITION_DATE);
END$$
DELIMITER ;
DELIMITER $$
CREATE PROCEDURE `partition_verify`(SCHEMANAMEVARCHAR(64), TABLENAME VARCHAR(64), HOURLYINTERVAL INT(11))
BEGIN
DECLARE PARTITION_NAME VARCHAR(16);
DECLARE RETROWS INT(11);
DECLARE FUTURE_TIMESTAMP TIMESTAMP;
/*
* Check if any partitions exist for the given SCHEMANAME.TABLENAME.
*/
SELECT COUNT(1) INTO RETROWS
FROM information_schema.partitions
WHERE table_schema = SCHEMANAME AND table_name = TABLENAME ANDpartition_name IS NULL;
/*
* If partitions do not exist, go ahead and partition the table
*/
IFRETROWS = 1 THEN
/*
* Take the current date at 00:00:00 and add HOURLYINTERVAL to it. This is the timestamp below which we willstore values.
* We begin partitioning based on the beginning of a day. This is because we don't want to generate arandom partition
* that won't necessarily fall in line with the desired partition naming(ie: if the hour interval is 24 hours, we could
* end up creating a partition now named "p201403270600" whenall other partitions will be like "p201403280000").
*/
SET FUTURE_TIMESTAMP = TIMESTAMPADD(HOUR, HOURLYINTERVAL,CONCAT(CURDATE(), " ", '00:00:00'));
SET PARTITION_NAME = DATE_FORMAT(CURDATE(), 'p%Y%m%d%H00');
-- Create the partitioning query
SET @__PARTITION_SQL = CONCAT("ALTER TABLE ", SCHEMANAME,".", TABLENAME, " PARTITION BY RANGE(`clock`)");
SET @__PARTITION_SQL = CONCAT(@__PARTITION_SQL, "(PARTITION ",PARTITION_NAME, " VALUES LESS THAN (",UNIX_TIMESTAMP(FUTURE_TIMESTAMP), "));");
-- Run the partitioning query
PREPARE STMT FROM @__PARTITION_SQL;
EXECUTE STMT;
DEALLOCATE PREPARE STMT;
END IF;
END$$
DELIMITER ;
DELIMITER $$
CREATE PROCEDURE`partition_maintenance_all`(SCHEMA_NAME VARCHAR(32))
BEGIN
CALL partition_maintenance(SCHEMA_NAME, 'history', 90, 24, 14);
CALL partition_maintenance(SCHEMA_NAME, 'history_log', 90, 24, 14);
CALL partition_maintenance(SCHEMA_NAME, 'history_str', 90, 24, 14);
CALL partition_maintenance(SCHEMA_NAME, 'history_text', 90, 24, 14);
CALLpartition_maintenance(SCHEMA_NAME, 'history_uint', 90, 24, 14);
CALL partition_maintenance(SCHEMA_NAME, 'trends', 730, 24, 14);
CALL partition_maintenance(SCHEMA_NAME, 'trends_uint', 730, 24, 14);
END$$
DELIMITER ;The above contains the stored procedure to create the partition, copy the above into partition. sql, and execute as follows:
mysql -uzabbix -pzabbix zabbix < partition.sqlb, add crontable, execute at 01:01 every day, as follows:
crontab -l > crontab.txt
cat >> crontab.txt <<EOF
#zabbix partition_maintenance
01 01 * * * mysql -uzabbix -pzabbix zabbix -e"CALL partition_maintenance_all('zabbix')" &>/dev/null
EOF
cat crontab.txt |crontabNote: The password part of the zabbix user of mysql is configured according to the actual environment
c, execute once first (use nohup because of the long time for first execution), as follows:
nohup mysql -uzabbix -pzabbix zabbix -e "CALLpartition_maintenance_all('zabbix')" &> /root/partition.log&Note: Observe the output of/root/partition. log
d, View Results
Log in to mysql and view history and other tables as follows:
MariaDB [zabbix]> showcreate table history
| history | CREATE TABLE `history` (
`itemid` bigint(20) unsigned NOT NULL,
`clock`int(11) NOT NULL DEFAULT '0',
`value`double(16,4) NOT NULL DEFAULT '0.0000',
`ns`int(11) NOT NULL DEFAULT '0',
KEY`history_1` (`itemid`,`clock`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
/*!50100 PARTITION BY RANGE (`clock`)
(PARTITION p201708280000 VALUES LESS THAN(1503936000) ENGINE = InnoDB,
PARTITION p201708290000 VALUES LESS THAN(1504022400) ENGINE = InnoDB,
PARTITION p201708300000 VALUES LESS THAN(1504108800) ENGINE = InnoDB,
PARTITION p201708310000 VALUES LESS THAN(1504195200) ENGINE = InnoDB,
PARTITION p201709010000 VALUES LESS THAN(1504281600) ENGINE = InnoDB,
PARTITION p201709020000 VALUES LESS THAN(1504368000) ENGINE = InnoDB,
PARTITION p201709030000 VALUES LESS THAN(1504454400) ENGINE = InnoDB,
PARTITION p201709040000 VALUES LESS THAN(1504540800) ENGINE = InnoDB,
PARTITION p201709050000 VALUES LESS THAN(1504627200) ENGINE = InnoDB,
PARTITION p201709060000 VALUES LESS THAN(1504713600) ENGINE = InnoDB,
PARTITION p201709070000 VALUES LESS THAN(1504800000) ENGINE = InnoDB,
PARTITION p201709080000 VALUES LESS THAN(1504886400) ENGINE = InnoDB,
PARTITION p201709090000 VALUES LESS THAN(1504972800) ENGINE = InnoDB,
PARTITION p201709100000 VALUES LESS THAN(1505059200) ENGINE = InnoDB,
PARTITION p201709110000 VALUES LESS THAN(1505145600) ENGINE = InnoDB) */ |A large number of PARTITION fields were found, indicating that the configuration is correct. Pay attention to the Slow Query of Mysql. By the second day of operation, Slow Query will almost appear. At this time, the response speed of Dashboard of Zabbix should be very smooth.