zabbix 数据库分区分表,彻底解决烦人的Zabbix housekeeper processes more than 75% busy
Zabbix的内务管理流程负责删除旧的趋势和历史数据。使用 SQL 删除查询从数据库中删除旧数据可能会对数据库性能产生负面影响。因此,我们中的许多人都收到了烦人的警报“”。Zabbix housekeeper processes more than 75% busy这个问题可以通过数据库分区轻松解决。分区为每小时或每天创建表,并在不再需要时删除它们。SQL DROP 比 DELETE 语句更有效。
Zabbix从主机收集数据,并使用历史记录和趋势表将其存储在数据库中。Zabbix历史记录保留原始数据(Zabbix收集的每个值),趋势存储合并的每小时数据,平均为最小值,平均值和最大值。
Zabbix的内务管理流程负责删除旧的趋势和历史数据。使用 SQL 删除查询从数据库中删除旧数据可能会对数据库性能产生负面影响。因此,我们中的许多人都收到了烦人的警报“”。Zabbix housekeeper processes more than 75% busy
这个问题可以通过数据库分区轻松解决。分区为每小时或每天创建表,并在不再需要时删除它们。SQL DROP 比 DELETE 语句更有效。
在我们继续之前,请备份Zabbix数据库,但如果安装是新的,则不需要备份。
在数据库服务器上下载并解压缩 SQL 脚本 “”(使用 wget 或 curl 工具):zbx_db_partitiong.sql
curl -O https://bestmonitoringtools.com/dl/zbx_db_partitiong.tar.gz tar -zxvf zbx_db_partitiong.tar.gz
脚本 “” 配置为保留 7 天的历史数据和 365 天的趋势数据 – 如果您可以接受这些设置,请转到步骤 2。zbx_db_partitiong.sql
但是,如果要更改趋势或历史记录的天数,请打开文件“zbx_db_partitiong.sql”,更改如下图所示的设置,然后保存文件。
步骤 2:使用 SQL 脚本创建分区过程
运行脚本的语法是“mysql -u ‘<db_username>’ -p’<db_password>’ <zb_database_name> < zbx_db_partitiong.sql”。
现在,使用您的Zabbix数据库名称,用户名和密码运行它以创建分区过程:
mysql -u ‘zabbix’ -p’zabbixDBpass’ zabbix < zbx_db_partitiong.sql
脚本将在新的Zabbix安装上非常快速地创建MySQL分区过程,但在大型数据库上,这可能会持续数小时。
步骤 3:自动运行分区过程
我们已经创建了分区过程,但在我们运行它们之前,它们不会执行任何操作!
此步骤是最重要的,因为必须使用分区过程定期(每天)删除和创建分区!
别担心,您不必手动执行此操作。我们可以使用两种工具来完成此类任务:MySQL 事件调度程序或 Crontab – 选择您喜欢的任何工具。
配置 MySQL 事件调度程序或 Crontab 时要小心。如果您配置错误,Zabbix将停止收集数据!您会注意到,通过空图形和Zabbix日志文件中的错误“[Z3005]查询失败:[1526]表没有值…的分区”。
选项 1:使用 MySQL 事件调度程序自动管理分区(推荐)
默认情况下,MySQL 事件调度程序处于禁用状态。您需要通过在MySQL配置文件中的“[mysqld]”行之后设置“event_scheduler=ON”来启用它。
[mysqld]
event_scheduler = ON
进行更改后,请重新启动MySQL服务器以使设置生效!
sudo systemctl restart mysql
好!应该启用MySQL事件调度程序,让我们使用以下命令进行检查:
root@dbserver:~ $ mysql -u ‘zabbix’ -p’zabbixDBpass’ zabbix -e “SHOW VARIABLES LIKE ‘event_scheduler’;”
±----------------±------+
| Variable_name | Value |
±----------------±------+
| event_scheduler | ON |
±----------------±------+
现在我们可以创建一个事件,该事件将每 12 小时运行一次过程“partition_maintenance_all”。
mysql -u ‘zabbix’ -p’zabbixDBpass’ zabbix -e “CREATE EVENT zbx_partitioning ON SCHEDULE EVERY 12 HOUR DO CALL partition_maintenance_all(‘zabbix’);”
12 小时后,使用以下命令检查事件是否已成功执行。
mysql -u ‘zabbix’ -p’zabbixDBpass’ zabbix -e “SELECT * FROM INFORMATION_SCHEMA.events\G”
选项 2:使用 Crontab 自动管理分区
如果您无法使用MySQL事件调度程序,Crontab是一个不错的选择。使用命令“sudo crontab -e”打开crontab文件,并通过在文件中的任何位置添加以下行来添加用于对Zabbix MySQL数据库进行分区的作业(每天凌晨03:30):
30 03 * * * /usr/bin/mysql -u ‘zabbix’ -p’zabbixDBpass’ zabbix -e “CALL partition_maintenance_all(‘zabbix’);” > /tmp/CronDBpartitiong.log 2>&1
保存并关闭文件。
Cron 将每天执行操作(删除旧表并创建新表)并将所有内容记录在文件 “” 中。/tmp/CronDBpartitiong.log
但是,如果您不想等待,请立即从终端运行命令:
root@dbserver:~ $ mysql -u ‘zabbix’ -p’zabbixDBpass’ zabbix -e “CALL partition_maintenance_all(‘zabbix’);” ±----------------------------------------------------------+ | msg | ±----------------------------------------------------------+ | partition_create(zabbix,history,p201910150000,1571180400) | ±----------------------------------------------------------+ ±----------------------------------------------------------+
并在之后检查分区状态:
root@dbserver:~ $ mysql -u ‘zabbix’ -p’zabbixDBpass’ zabbix -e “show create table history\G” Table: history Create Table: CREATE TABLE history ( itemid bigint(20) unsigned NOT NULL, clock int(11) NOT NULL DEFAULT ‘0’, value double(16,4) NOT NULL DEFAULT ‘0.0000’, ns int(11) NOT NULL DEFAULT ‘0’, KEY history_1 (itemid,clock) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin /*!50100 PARTITION BY RANGE (clock) (PARTITION p201910140000 VALUES LESS THAN (1571094000) ENGINE = InnoDB, PARTITION p201910150000 VALUES LESS THAN (1571180400) ENGINE = InnoDB, PARTITION p201910160000 VALUES LESS THAN (1571266800) ENGINE = InnoDB) */
正如您在输出中看到的,我们为历史记录表创建了 3 个分区。
步骤4:在Zabbix前端配置内务管理
在Zabbix前端配置内务管理,如下图所示。
如果图片不言自明,以下是在Zabbix前端配置内务管理的步骤:
- 导航到“家政”部分:“管理”→“常规”→“家政”;
- 从“历史记录和趋势”部分下的“启用内部内务管理”中删除复选标记;
- 在“历史记录和趋势”部分下的“_覆盖项目趋势周期”_上打勾;
- 在“历史和趋势”部分下为趋势和历史记录定义“数据存储期”的天数(必须与数据库分区中配置的天数相同 – 如果尚未更改脚本中的默认设置,则历史记录应为 7 天,趋势应为 365 天);
- 单击“更新”按钮。
大功告成!请记住,分区将根据您在分区过程中配置的内容删除历史记录和趋势表。例如,如果已配置为保留 7 天的历史记录,则分区将在第 8 天开始删除历史记录。之后,它将每天删除一个历史记录表,以便数据库始终具有 7 天的历史数据。趋势数据也是如此,如果您配置为保留 365 天的趋势数据,则只有在 365 天后,它才会开始删除旧的趋势表。
步骤 5:更改分区设置(历史记录和趋势的天数)
有时,您最初可能会为Zabbix数据库设置太多的历史记录和趋势天数,因此磁盘空间填满得太快。或者相反的情况,您没有为历史记录或趋势配置足够的天数。那怎么办?
您无需再次运行脚本,只需创建一个将运行的新过程而不是旧过程即可。
a) 创建新的分区过程
连接到 MySQL/MariaDB 服务器:
mysql -u ‘zabbix’ -p’zabbixDBpass’ zabbix
创建一个新过程,但根据您的需要更改趋势和历史记录的天数,我将为历史记录设置 30 天,为趋势设置 400 天:
DELIMITER
C
R
E
A
T
E
P
R
O
C
E
D
U
R
E
p
a
r
t
i
t
i
o
n
m
a
i
n
t
e
n
a
n
c
e
a
l
l
3
0
a
n
d
400
(
S
C
H
E
M
A
N
A
M
E
V
A
R
C
H
A
R
(
32
)
)
B
E
G
I
N
C
A
L
L
p
a
r
t
i
t
i
o
n
m
a
i
n
t
e
n
a
n
c
e
(
S
C
H
E
M
A
N
A
M
E
,
′
h
i
s
t
o
r
y
′
,
30
,
24
,
3
)
;
C
A
L
L
p
a
r
t
i
t
i
o
n
m
a
i
n
t
e
n
a
n
c
e
(
S
C
H
E
M
A
N
A
M
E
,
′
h
i
s
t
o
r
y
l
o
g
′
,
30
,
24
,
3
)
;
C
A
L
L
p
a
r
t
i
t
i
o
n
m
a
i
n
t
e
n
a
n
c
e
(
S
C
H
E
M
A
N
A
M
E
,
′
h
i
s
t
o
r
y
s
t
r
′
,
30
,
24
,
3
)
;
C
A
L
L
p
a
r
t
i
t
i
o
n
m
a
i
n
t
e
n
a
n
c
e
(
S
C
H
E
M
A
N
A
M
E
,
′
h
i
s
t
o
r
y
t
e
x
t
′
,
30
,
24
,
3
)
;
C
A
L
L
p
a
r
t
i
t
i
o
n
m
a
i
n
t
e
n
a
n
c
e
(
S
C
H
E
M
A
N
A
M
E
,
′
h
i
s
t
o
r
y
u
i
n
t
′
,
30
,
24
,
3
)
;
C
A
L
L
p
a
r
t
i
t
i
o
n
m
a
i
n
t
e
n
a
n
c
e
(
S
C
H
E
M
A
N
A
M
E
,
′
t
r
e
n
d
s
′
,
400
,
24
,
3
)
;
C
A
L
L
p
a
r
t
i
t
i
o
n
m
a
i
n
t
e
n
a
n
c
e
(
S
C
H
E
M
A
N
A
M
E
,
′
t
r
e
n
d
s
u
i
n
t
′
,
400
,
24
,
3
)
;
E
N
D
CREATE PROCEDURE partition_maintenance_all_30and400(SCHEMA_NAME VARCHAR(32)) BEGIN CALL partition_maintenance(SCHEMA_NAME, 'history', 30, 24, 3); CALL partition_maintenance(SCHEMA_NAME, 'history_log', 30, 24, 3); CALL partition_maintenance(SCHEMA_NAME, 'history_str', 30, 24, 3); CALL partition_maintenance(SCHEMA_NAME, 'history_text', 30, 24, 3); CALL partition_maintenance(SCHEMA_NAME, 'history_uint', 30, 24, 3); CALL partition_maintenance(SCHEMA_NAME, 'trends', 400, 24, 3); CALL partition_maintenance(SCHEMA_NAME, 'trends_uint', 400, 24, 3); END
CREATEPROCEDUREpartitionmaintenanceall30and400(SCHEMANAMEVARCHAR(32))BEGINCALLpartitionmaintenance(SCHEMANAME,′history′,30,24,3);CALLpartitionmaintenance(SCHEMANAME,′historylog′,30,24,3);CALLpartitionmaintenance(SCHEMANAME,′historystr′,30,24,3);CALLpartitionmaintenance(SCHEMANAME,′historytext′,30,24,3);CALLpartitionmaintenance(SCHEMANAME,′historyuint′,30,24,3);CALLpartitionmaintenance(SCHEMANAME,′trends′,400,24,3);CALLpartitionmaintenance(SCHEMANAME,′trendsuint′,400,24,3);END DELIMITER ;
b) 更新 MySQL 事件调度程序或 Crontab
我们已经在上一步中创建了分区过程,但它尚未激活!现在我们必须用新程序替换旧过程,该过程将定期删除和添加分区。选择以下两个选项之一,具体取决于您在Zabbix实例上配置的内容。
选项 1:更新 MySQL 事件调度程序
如果按照本教程创建了事件计划程序,请使用此命令将旧过程替换为新过程。
mysql -u 'zabbix' -p'zabbixDBpass' zabbix -e "ALTER EVENT zbx_partitioning ON SCHEDULE EVERY 12 HOUR DO CALL partition_maintenance_all_30and400('zabbix');"
选项 2:更新 Crontab
对于那些使用 Crontab 的用户,使用命令“sudo crontab -e”打开 crontab 文件,注释掉旧的过程作业,然后添加一个新
# old procedure, still exists in the database so it can be used if needed # 30 03 * * * /usr/bin/mysql -u 'zabbix' -p'zabbixDBpass' zabbix -e "CALL partition_maintenance_all('zabbix');" > /tmp/CronDBpartitiong.log 2>&1 30 03 * * * /usr/bin/mysql -u 'zabbix' -p'zabbixDBpass' zabbix -e "CALL partition_maintenance_all_30and400('zabbix');" > /tmp/CronDBpartitiong.log 2>&1
保存更改并退出 Crontab。
第 6 步:有关的信息 Zabbix 分区脚本
本指南中使用的Zabbix分区SQL脚本包含以下分区过程:
DELIMITER $$ CREATE PROCEDURE `partition_create`(SCHEMANAME varchar(64), TABLENAME varchar(64), PARTITIONNAME varchar(64), CLOCK int) BEGIN /* SCHEMANAME = The DB schema in which to make changes TABLENAME = The table with partitions to potentially delete PARTITIONNAME = The name of the partition to create */ /* Verify that the partition does not already exist */ DECLARE RETROWS INT; SELECT COUNT(1) INTO RETROWS FROM information_schema.partitions WHERE table_schema = SCHEMANAME AND table_name = TABLENAME AND partition_description >= CLOCK; IF RETROWS = 0 THEN /* 1. Print a message indicating that a partition was created. 2. Create the SQL to create the partition. 3. Execute the SQL from #2. */ SELECT CONCAT( "partition_create(", SCHEMANAME, ",", TABLENAME, ",", PARTITIONNAME, ",", CLOCK, ")" ) AS msg; SET @sql = CONCAT( 'ALTER TABLE ', SCHEMANAME, '.', TABLENAME, ' ADD PARTITION (PARTITION ', PARTITIONNAME, ' VALUES LESS THAN (', CLOCK, '));' ); PREPARE STMT FROM @sql; EXECUTE STMT; DEALLOCATE PREPARE STMT; END IF; END$$ DELIMITER ; DELIMITER $$ CREATE PROCEDURE `partition_drop`(SCHEMANAME VARCHAR(64), TABLENAME VARCHAR(64), DELETE_BELOW_PARTITION_DATE BIGINT) BEGIN /* SCHEMANAME = The DB schema in which to make changes TABLENAME = The table with partitions to potentially delete DELETE_BELOW_PARTITION_DATE = Delete any partitions with names that are dates older than this one (yyyy-mm-dd) */ DECLARE done INT DEFAULT FALSE; DECLARE drop_part_name VARCHAR(16); /* Get a list of all the partitions that are older than the date in DELETE_BELOW_PARTITION_DATE. All partitions are prefixed with a "p", so use SUBSTRING TO get rid of that character. */ DECLARE myCursor CURSOR FOR SELECT partition_name FROM information_schema.partitions WHERE table_schema = SCHEMANAME AND table_name = TABLENAME AND CAST(SUBSTRING(partition_name FROM 2) AS UNSIGNED) < DELETE_BELOW_PARTITION_DATE; DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = TRUE; /* Create the basics for when we need to drop the partition. Also, create @drop_partitions to hold a comma-delimited list of all partitions that should be deleted. */ SET @alter_header = CONCAT("ALTER TABLE ", SCHEMANAME, ".", TABLENAME, " DROP PARTITION "); SET @drop_partitions = ""; /* Start looping through all the partitions that are too old. */ OPEN myCursor; read_loop: LOOP FETCH myCursor INTO drop_part_name; IF done THEN LEAVE read_loop; END IF; SET @drop_partitions = IF(@drop_partitions = "", drop_part_name, CONCAT(@drop_partitions, ",", drop_part_name)); END LOOP; IF @drop_partitions != "" THEN /* 1. Build the SQL to drop all the necessary partitions. 2. Run the SQL to drop the partitions. 3. Print out the table partitions that were deleted. */ SET @full_sql = CONCAT(@alter_header, @drop_partitions, ";"); PREPARE STMT FROM @full_sql; EXECUTE STMT; DEALLOCATE PREPARE STMT; SELECT CONCAT(SCHEMANAME, ".", TABLENAME) AS `table`, @drop_partitions AS `partitions_deleted`; ELSE /* No partitions are being deleted, so print out "N/A" (Not applicable) to indicate that no changes were made. */ SELECT CONCAT(SCHEMANAME, ".", TABLENAME) AS `table`, "N/A" AS `partitions_deleted`; END IF; END$$ DELIMITER ; DELIMITER $$ CREATE PROCEDURE `partition_maintenance`(SCHEMA_NAME VARCHAR(32), TABLE_NAME VARCHAR(32), KEEP_DATA_DAYS INT, HOURLY_INTERVAL INT, CREATE_NEXT_INTERVALS INT) BEGIN DECLARE OLDER_THAN_PARTITION_DATE VARCHAR(16); DECLARE PARTITION_NAME VARCHAR(16); DECLARE OLD_PARTITION_NAME VARCHAR(16); DECLARE LESS_THAN_TIMESTAMP INT; DECLARE CUR_TIME INT; CALL partition_verify(SCHEMA_NAME, TABLE_NAME, HOURLY_INTERVAL); SET CUR_TIME = UNIX_TIMESTAMP(DATE_FORMAT(NOW(), '%Y-%m-%d 00:00:00')); SET @__interval = 1; create_loop: LOOP IF @__interval > CREATE_NEXT_INTERVALS THEN LEAVE create_loop; END IF; SET LESS_THAN_TIMESTAMP = CUR_TIME + (HOURLY_INTERVAL * @__interval * 3600); SET PARTITION_NAME = FROM_UNIXTIME(CUR_TIME + HOURLY_INTERVAL * (@__interval - 1) * 3600, 'p%Y%m%d%H00'); IF(PARTITION_NAME != OLD_PARTITION_NAME) THEN CALL partition_create(SCHEMA_NAME, TABLE_NAME, PARTITION_NAME, LESS_THAN_TIMESTAMP); END IF; SET @__interval=@__interval+1; SET OLD_PARTITION_NAME = PARTITION_NAME; END LOOP; SET OLDER_THAN_PARTITION_DATE=DATE_FORMAT(DATE_SUB(NOW(), INTERVAL KEEP_DATA_DAYS DAY), '%Y%m%d0000'); CALL partition_drop(SCHEMA_NAME, TABLE_NAME, OLDER_THAN_PARTITION_DATE); END$$ DELIMITER ; DELIMITER $$ CREATE PROCEDURE `partition_verify`(SCHEMANAME VARCHAR(64), TABLENAME VARCHAR(64), HOURLYINTERVAL INT(11)) BEGIN DECLARE PARTITION_NAME VARCHAR(16); DECLARE RETROWS INT(11); DECLARE FUTURE_TIMESTAMP TIMESTAMP; /* * Check if any partitions exist for the given SCHEMANAME.TABLENAME. */ SELECT COUNT(1) INTO RETROWS FROM information_schema.partitions WHERE table_schema = SCHEMANAME AND table_name = TABLENAME AND partition_name IS NULL; /* * If partitions do not exist, go ahead and partition the table */ IF RETROWS = 1 THEN /* * Take the current date at 00:00:00 and add HOURLYINTERVAL to it. This is the timestamp below which we will store values. * We begin partitioning based on the beginning of a day. This is because we don't want to generate a random partition * that won't necessarily fall in line with the desired partition naming (ie: if the hour interval is 24 hours, we could * end up creating a partition now named "p201403270600" when all other partitions will be like "p201403280000"). */ SET FUTURE_TIMESTAMP = TIMESTAMPADD(HOUR, HOURLYINTERVAL, CONCAT(CURDATE(), " ", '00:00:00')); SET PARTITION_NAME = DATE_FORMAT(CURDATE(), 'p%Y%m%d%H00'); -- Create the partitioning query SET @__PARTITION_SQL = CONCAT("ALTER TABLE ", SCHEMANAME, ".", TABLENAME, " PARTITION BY RANGE(`clock`)"); SET @__PARTITION_SQL = CONCAT(@__PARTITION_SQL, "(PARTITION ", PARTITION_NAME, " VALUES LESS THAN (", UNIX_TIMESTAMP(FUTURE_TIMESTAMP), "));"); -- Run the partitioning query PREPARE STMT FROM @__PARTITION_SQL; EXECUTE STMT; DEALLOCATE PREPARE STMT; END IF; END$$ DELIMITER ; DELIMITER $$ CREATE PROCEDURE `partition_maintenance_all`(SCHEMA_NAME VARCHAR(32)) BEGIN CALL partition_maintenance(SCHEMA_NAME, 'history', 7, 24, 3); CALL partition_maintenance(SCHEMA_NAME, 'history_log', 7, 24, 3); CALL partition_maintenance(SCHEMA_NAME, 'history_str', 7, 24, 3); CALL partition_maintenance(SCHEMA_NAME, 'history_text', 7, 24, 3); CALL partition_maintenance(SCHEMA_NAME, 'history_uint', 7, 24, 3); CALL partition_maintenance(SCHEMA_NAME, 'trends', 365, 24, 3); CALL partition_maintenance(SCHEMA_NAME, 'trends_uint', 365, 24, 3); END$$ DELIMITER ;
更多推荐
所有评论(0)