场景：

  pt环境下研发反馈说redis集群down 掉了，报(error) CLUSTERDOWN The cluster is down

由于是在k8s 里运行redis-cluster，随便登录上其中一个节点去查询，发现在82这个节点上cluster_state状态是fail，那么问题就定位到82这台机器上，网上也搜索了一下问题原因好像是slots不连续造成的，那试着网上的办法去解决：

1.登录出现问题的那个节点，运行cluster info

10.42.230.82:6379> cluster info 
cluster_state:fail
cluster_slots_assigned:16380
cluster_slots_ok:16380
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:10
cluster_size:5
cluster_current_epoch:51
cluster_my_epoch:11
cluster_stats_messages_ping_sent:5190185
cluster_stats_messages_pong_sent:5617414
cluster_stats_messages_meet_sent:389904
cluster_stats_messages_fail_sent:52
cluster_stats_messages_auth-ack_sent:1
cluster_stats_messages_sent:11197556
cluster_stats_messages_ping_received:5231452
cluster_stats_messages_pong_received:5579773
cluster_stats_messages_meet_received:385962
cluster_stats_messages_fail_received:6
cluster_stats_messages_auth-req_received:1
cluster_stats_messages_received:11197194

 执行cluster slots 查看槽点分布情况

10.42.230.82:6379> get nodes
(error) CLUSTERDOWN The cluster is down
10.42.230.82:6379> get node
(error) CLUSTERDOWN The cluster is down
10.42.230.82:6379> get name
(error) CLUSTERDOWN The cluster is down
10.42.230.82:6379> get names
(error) CLUSTERDOWN The cluster is down
10.42.230.82:6379> cluster slots
1) 1) (integer) 3278
   2) (integer) 6553
   3) 1) "10.42.209.18"
      2) (integer) 6379
      3) "eb3160f72b96b22b95963dfdde346b4a10bac834"
   4) 1) "10.42.230.80"
      2) (integer) 6379
      3) "46daeb573cd80ff2a8724351b3e16a986b0ffdaa"
2) 1) (integer) 9831
   2) (integer) 13106
   3) 1) "10.42.230.79"
      2) (integer) 6379
      3) "3cfe4da28c8e3c951e0ea8f1353c2657f11f56f9"
   4) 1) "10.42.34.111"
      2) (integer) 6379
      3) "34db05f361696fc0d23af96fa2a3a9c22c633f16"
3) 1) (integer) 1
   2) (integer) 3276
   3) 1) "10.42.209.17"
      2) (integer) 6379
      3) "f9ebe7de3aa3ebe229c6cdef8fc9cd4c52c006e6"
   4) 1) "10.42.230.81"
      2) (integer) 6379
      3) "5a3271993366e0023fd9aa53049f65a120d8990b"
4) 1) (integer) 6554
   2) (integer) 9829
   3) 1) "10.42.230.82"
      2) (integer) 6379
      3) "668d00003902026832f71d531b7ed7f033f7f117"
   4) 1) "10.42.209.16"
      2) (integer) 6379
      3) "cf044f2bd5d8f22d210ce401a350aa1d719500f3"
5) 1) (integer) 13108
   2) (integer) 16383
   3) 1) "10.42.217.210"
      2) (integer) 6379
      3) "66318424adc132494c57124c6748c06e2abcccb6"
   4) 1) "10.42.209.15"
      2) (integer) 6379
      3) "49140eb3128da7ed80c8c1355a375d109a59fe2b"

统计一下slots,结果如下：

10.42.209.18  slots: 3278~6553 
10.40.230.82  slots:6554~9829
10.42.230.79  slots: 9831~13106  
10.42.217.210 slots:13108~16383

发现slots，缺少0 3277 9830 13107 4个槽点，下面按照网上说的办法在出问题的机器上手动分配一下：

10.42.230.82:6379> CLUSTER ADDSLOTS 0 3277 9830 13107
OK

再次查看集群状态，发现状态已经ok，让测试那边也验证了一下一切正常。

10.42.230.82:6379> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:10
cluster_size:5
cluster_current_epoch:51
cluster_my_epoch:11
cluster_stats_messages_ping_sent:5192739
cluster_stats_messages_pong_sent:5620047
cluster_stats_messages_meet_sent:390015
cluster_stats_messages_fail_sent:52
cluster_stats_messages_auth-ack_sent:1
cluster_stats_messages_sent:11202854
cluster_stats_messages_ping_received:5234004
cluster_stats_messages_pong_received:5582438
cluster_stats_messages_meet_received:386043
cluster_stats_messages_fail_received:6
cluster_stats_messages_auth-req_received:1
cluster_stats_messages_update_received:13
cluster_stats_messages_received:11202505

到此问题已经解决了，但是为啥会出现这种情况呢，redis环境也没有异常断电重启之类的操作。