一、环境&配置

鲲鹏服务器 128c 512G *6

每台主机普通SSD *5

Kylin V10（arm）

主机TiDBPDTiKV
ecs0014 node1node4 node
ecs0024 node1node4 node
ecs0034 node1node4 node
ecs0044 node4 node
ecs0054 node4 node
ecs0064 node4 node

每个TiKV对应独立mount的文件系统（普通SSD）

总体配置：24TiDB 3PD 24TiKV

二、 优化

2.1、系统层面

1.1、TiDB/PD/TiKV绑核

        128c，4个node，通过参数numa_node各绑一个node

image

 注：

1、理论上PD需要单独服务器部署（后期有机器再测：）

2、网络中断全绑一个node上（不用和tikv/tidb分开）

1.2、HAproxy绑核

        HAproxy部署在单独的主机上，通过map参数绑核

global                                     # 全局配置。
   log         127.0.0.1 local2            # 定义全局的 syslog 服务器，最多可以定义两个。
   chroot      /var/lib/haproxy            # 更改当前目录并为启动进程设置超级用户权限，从而提高安全性。
   pidfile     /var/run/haproxy.pid        # 将 HAProxy 进程的 PID 写入 pidfile。
   maxconn     4000                        # 每个 HAProxy 进程所接受的最大并发连接数。
   user        haproxy                     # 同 UID 参数。
   group       haproxy                     # 同 GID 参数，建议使用专用用户组。
   nbproc      64                          # 在后台运行时创建的进程数。在启动多个进程转发请求时，确保该值足够大，保证 HAProxy 不会成为瓶颈。
   daemon                                  # 让 HAProxy 以守护进程的方式工作于后台，等同于命令行参数“-D”的功能。当然，也可以在命令行中用“-db”参数将其禁用。
   stats socket /var/lib/haproxy/stats     # 统计信息保存位置。
   cpu-map      1       0
   cpu-map      2       1
   cpu-map      3       2
   cpu-map      4       3
   cpu-map      5       4
   cpu-map      6       5
   cpu-map      7       6
   cpu-map      8       7
   cpu-map      9       8
   cpu-map      10      9
   cpu-map      11      10
   cpu-map      12      11
   cpu-map      13      12
   cpu-map      14      13
   cpu-map      15      14
   cpu-map      16      15
.....

1.3、sysbench 和 tiup 绑核运行

# vi oltp_point_select.sh
nohup numactl -c 0-31 sysbench ... &
nohup numactl -c 32-63 sysbench ... &
nohup numactl -c 64-95 sysbench ... &
nohup numactl -c 96-127 sysbench ... &
<最后可加入文本处理语句，直接查看tps/qps>
<例：4000并发就每个sysbench启1000线程>

tiup bench tpcc因为只测1000线程，所以只需要绑一个node；绑多个node会造成测试结果下降

2.2、数据库层面

Tidb:
	log.level: "error"
	prepared-plan-cache.enabled: true
	tikv-client.max-batch-wait-time: 2000000
	performance.txn-total-size-limit: 10737418240  
	txn-local-latches.enabled: true
	tikv-client.grpc-connection-count: 10  
tikv:
	pessimistic-txn.pipelined: true
	raftdb.allow-concurrent-memtable-write: true
	raftdb.max-background-jobs: 16
	raftstore.sync-log: false
	raftstore.apply-max-batch-size: 2048
	raftstore.apply-pool-size: 5
	raftstore.store-max-batch-size: 2048
	raftstore.store-pool-size: 5
	readpool.storage.high-concurrency: 30
	readpool.storage.normal-concurrency: 30
	readpool.storage.low-concurrency: 30
	readpool.unified.max-thread-count: 20
	readpool.unified.min-thread-count: 5
	rocksdb.max-background-jobs: 15
	server.grpc-concurrency: 10
	storage.scheduler-worker-pool-size: 20
	server.enable-request-batch: false


Tidb全局：
	set global tidb_hashagg_final_concurrency=1;
	set global tidb_hashagg_partial_concurrency=1;
	set global tidb_enable_async_commit = 1;
	set global tidb_enable_1pc = 1;
	set global tidb_guarantee_linearizability = 0;
	set global tidb_enable_clustered_index = 1;

    set global tidb_enable_tso_follower_proxy=true;
    //5.3加入的新特性，高并发场景下点查延迟大幅度降低

三、部分测试结果

3.1、sysbench

32张表，单表1亿，rand-type=uniform，测试先查后写

sysbench 1.0.20，github源码编译

场景并发数（threads）Tidb v5.3.0
point_select100tps229820.37
95th 延迟0.49
300tps789232
95th 延迟1.1
1000tps1101896.89
95th 延迟2.53
4000tps1438272.43
95th 延迟9.06
6000tps1604941.51
95th 延迟11.92
read_only100tps8591.23
queries137460
95th 延迟17.79
300tps18403.8
queries294461
95th 延迟25.28
1000tps26467
queries423473
95th 延迟52.89
4000tps21259.8
queries340157
95th 延迟493.24
6000tps22623
queries361968
95th 延迟679.01
write_only100tps7910.32
queries47461.9
95th 延迟28.93
300tps18172.2
queries109033
95th 延迟35.43
1000tps21846.1
queries131076
95th 延迟24.6
4000tps46397.9
queries278387
95th 延迟202.32
6000tps46103.3
queries276620
95th 延迟324.68
read_write100tps4590.59
queries91811.6
95th 延迟32.53
300tps10800.2
queries216004
95th 延迟44.57
1000tps

（sysbench频繁

     core dump）

queries
95th 延迟
4000tps
queries
95th 延迟
6000tps
queries
95th 延迟
update_index100tps19975.4
95th 延迟10.46
300tps39277.5
95th 延迟17.71
1000tps71013.4
95th 延迟33.72
4000tps116711
95th 延迟94.13
6000tps121850
95th 延迟134.9
update_no_index100tps30769.6
95th 延迟9.9
300tps66091.5
95th 延迟14.27
1000tps151905
95th 延迟17.95
4000tps291798
95th 延迟37.39
6000tps301261
95th 延迟49.43
select_random_point100tps43797.6
95th 延迟7.2
300tps85351.3
95th 延迟11.14
1000tps117452
95th 延迟18.61
4000tps138922
95th 延迟48.34
6000tps139389
95th 延迟69.29
select_random_point100tps47080.2
95th 延迟6.49
300tps99325.3
95th 延迟10.89
1000tps169312
95th 延迟17.32
4000tps295689
95th 延迟38.94
6000tps320196
95th 延迟55.82

3.2、TPCC

tiup bench tpcc ，1000线程， numactl单node， 5000仓，10min

[Summary] DELIVERY - Takes(s): 599.6, Count: 172248, TPM: 17236.9, Sum(ms): 48903503.2, Avg(ms): 284.0, 50th(ms): 184.5, 90th(ms): 637.5, 95th(ms): 838.9, 99th(ms): 1409.3, 99.9th(ms): 2281.7, Max(ms): 4563.4
[Summary] DELIVERY_ERR - Takes(s): 599.6, Count: 69, TPM: 6.9, Sum(ms): 18272.0, Avg(ms): 265.5, 50th(ms): 176.2, 90th(ms): 520.1, 95th(ms): 671.1, 99th(ms): 1409.3, 99.9th(ms): 1744.8, Max(ms): 1744.8
[Summary] NEW_ORDER - Takes(s): 599.7, Count: 1936102, TPM: 193711.0, Sum(ms): 283078092.3, Avg(ms): 146.3, 50th(ms): 83.9, 90th(ms): 335.5, 95th(ms): 503.3, 99th(ms): 872.4, 99.9th(ms): 1879.0, Max(ms): 5637.1
[Summary] NEW_ORDER_ERR - Takes(s): 599.7, Count: 264, TPM: 26.4, Sum(ms): 31434.9, Avg(ms): 119.3, 50th(ms): 29.4, 90th(ms): 369.1, 95th(ms): 419.4, 99th(ms): 503.3, 99.9th(ms): 704.6, Max(ms): 704.6
[Summary] ORDER_STATUS - Takes(s): 599.7, Count: 172011, TPM: 17209.3, Sum(ms): 1967141.7, Avg(ms): 11.4, 50th(ms): 7.3, 90th(ms): 21.0, 95th(ms): 28.3, 99th(ms): 71.3, 99.9th(ms): 226.5, Max(ms): 1275.1
[Summary] ORDER_STATUS_ERR - Takes(s): 599.7, Count: 3, TPM: 0.3, Sum(ms): 38.8, Avg(ms): 13.0, 50th(ms): 12.6, 90th(ms): 15.2, 95th(ms): 15.2, 99th(ms): 15.2, 99.9th(ms): 15.2, Max(ms): 15.2
[Summary] PAYMENT - Takes(s): 599.7, Count: 1846477, TPM: 184731.2, Sum(ms): 260149876.7, Avg(ms): 140.9, 50th(ms): 75.5, 90th(ms): 335.5, 95th(ms): 486.5, 99th(ms): 872.4, 99.9th(ms): 1811.9, Max(ms): 4160.7
[Summary] PAYMENT_ERR - Takes(s): 599.7, Count: 163, TPM: 16.3, Sum(ms): 19136.9, Avg(ms): 117.6, 50th(ms): 24.1, 90th(ms): 385.9, 95th(ms): 436.2, 99th(ms): 503.3, 99.9th(ms): 838.9, Max(ms): 838.9
[Summary] STOCK_LEVEL - Takes(s): 599.7, Count: 171785, TPM: 17186.8, Sum(ms): 3861185.1, Avg(ms): 22.5, 50th(ms): 13.1, 90th(ms): 29.4, 95th(ms): 50.3, 99th(ms): 234.9, 99.9th(ms): 637.5, Max(ms): 5100.3
[Summary] STOCK_LEVEL_ERR - Takes(s): 599.7, Count: 9, TPM: 0.9, Sum(ms): 1368.9, Avg(ms): 153.1, 50th(ms): 19.9, 90th(ms): 385.9, 95th(ms): 453.0, 99th(ms): 453.0, 99.9th(ms): 453.0, Max(ms): 453.0
tpmC: 193710.8, efficiency: 301.3%

四、其他

高度兼容ARM、快速迭代更新、良好的生态、快捷方便的文档、版本统一

                                        continue。。。

# 数据库 # kylin # arm
Logo
华为云开发者联盟

为开发者提供学习成长、分享交流、生态实践、资源工具等服务，帮助开发者快速成长。

更多推荐

cover

通过HPA+CronHPA组合应对业务复杂弹性伸缩场景

avatar 华为云开发者联盟
cover

FT-FMEA融合混沌演练，零售运营系统韧性架构在线验证实践

avatar 华为云开发者联盟
cover

如何使用Python和Plotly绘制3D图形

avatar 华为云开发者联盟