k8s使用glusterfs存储报错type 'features/utime' is not valid or not found on this machine

pods报错如下

Events:
  Type     Reason       Age    From                    Message
  ----     ------       ----   ----                    -------
  Normal   Scheduled    5m48s  default-scheduler       Successfully assigned default/auth-web-1-77f4f7cbcc-snp8c to 192.168.1.216
  Warning  FailedMount  5m47s  kubelet, 192.168.1.216  MountVolume.SetUp failed for volume "auth-web-home" : mount failed: mount failed: exit status 1
Mounting command: mount
Mounting arguments: -t glusterfs -o log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/auth-web-home/auth-web-1-77f4f7cbcc-snp8c-glusterfs.log,backup-volfile-servers=192.168.1.175:192.168.1.178:192.168.1.179,log-level=ERROR 192.168.1.175:k8s-glusterfs /var/lib/kubelet/pods/cb033ec1-6a62-11e9-b923-e0d55e1cf4c9/volumes/kubernetes.io~glusterfs/auth-web-home
Output: Mount failed. Please check the log file for more details.

 the following error information was pulled from the glusterfs log to help diagnose this issue:
[2019-04-29 09:40:16.070806] E [MSGID: 100026] [glusterfsd.c:2307:glusterfs_process_volfp] 0-: failed to construct the graph
[2019-04-29 09:40:16.071117] E [graph.c:982:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x12e) [0x5654d1b3138e] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x148) [0x5654d1b2bdc8] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7f7989cda68b] ) 0-graph: invalid argument: graph [Invalid argument]
  Warning  FailedMount  5m47s  kubelet, 192.168.1.216  MountVolume.SetUp failed for volume "auth-web-home" : mount failed: mount failed: exit status 1
Mounting command: mount
Mounting arguments: -t glusterfs -o log-level=ERROR,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/auth-web-home/auth-web-1-77f4f7cbcc-snp8c-glusterfs.log,backup-volfile-servers=192.168.1.175:192.168.1.178:192.168.1.179 192.168.1.175:k8s-glusterfs /var/lib/kubelet/pods/cb033ec1-6a62-11e9-b923-e0d55e1cf4c9/volumes/kubernetes.io~glusterfs/auth-web-home
Output: Mount failed. Please check the log file for more details.

根据报错信息,我们可以看出来挂载的时候他是指定的日志文件的,我们去对应的日志查看报错

[2019-04-29 09:40:16.070575] E [MSGID: 101002] [graph.y:212:volume_type] 0-parser: Volume 'k8s-glusterfs-utime', line 61: type 'features/utime' is not valid or not found on this machine
[2019-04-29 09:40:16.070647] E [MSGID: 101019] [graph.y:320:volume_end] 0-parser: "type" not specified for volume k8s-glusterfs-utime
[2019-04-29 09:40:16.070806] E [MSGID: 100026] [glusterfsd.c:2307:glusterfs_process_volfp] 0-: failed to construct the graph
[2019-04-29 09:40:16.071117] E [graph.c:982:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x12e) [0x5654d1b3138e] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x148) [0x5654d1b2bdc8] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7f7989cda68b] ) 0-graph: invalid argument: graph [Invalid argument]
[2019-04-29 09:40:16.712383] E [MSGID: 101002] [graph.y:212:volume_type] 0-parser: Volume 'k8s-glusterfs-utime', line 61: type 'features/utime' is not valid or not found on this machine
[2019-04-29 09:40:16.712464] E [MSGID: 101019] [graph.y:320:volume_end] 0-parser: "type" not specified for volume k8s-glusterfs-utime
[2019-04-29 09:40:16.712658] E [MSGID: 100026] [glusterfsd.c:2307:glusterfs_process_volfp] 0-: failed to construct the graph
[2019-04-29 09:40:16.713034] E [graph.c:982:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x12e) [0x55c7459fe38e] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x148) [0x55c7459f8dc8] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7f1a3cded68b] ) 0-graph: invalid argument: graph [Invalid argument]
[2019-04-29 09:40:17.868659] E [MSGID: 101002] [graph.y:212:volume_type] 0-parser: Volume 'k8s-glusterfs-utime', line 61: type 'features/utime' is not valid or not found on this machine
[2019-04-29 09:40:17.868718] E [MSGID: 101019] [graph.y:320:volume_end] 0-parser: "type" not specified for volume k8s-glusterfs-utime
[2019-04-29 09:40:17.868882] E [MSGID: 100026] [glusterfsd.c:2307:glusterfs_process_volfp] 0-: failed to construct the graph
[2019-04-29 09:40:17.869215] E [graph.c:982:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x12e) [0x5625d510738e] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x148) [0x5625d5101dc8] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7f578831768b] ) 0-graph: invalid argument: graph [Invalid argument]
[2019-04-29 09:40:20.068908] E [MSGID: 101002] [graph.y:212:volume_type] 0-parser: Volume 'k8s-glusterfs-utime', line 61: type 'features/utime' is not valid or not found on this machine
[2019-04-29 09:40:20.068976] E [MSGID: 101019] [graph.y:320:volume_end] 0-parser: "type" not specified for volume k8s-glusterfs-utime
[2019-04-29 09:40:20.069168] E [MSGID: 100026] [glusterfsd.c:2307:glusterfs_process_volfp] 0-: failed to construct the graph
[2019-04-29 09:40:20.069562] E [graph.c:982:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x12e) [0x55da540e838e] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x148) [0x55da540e2dc8] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7f1a9d62068b] ) 0-graph: invalid argument: graph [Invalid argument]
[2019-04-29 09:40:24.271395] E [MSGID: 101002] [graph.y:212:volume_type] 0-parser: Volume 'k8s-glusterfs-utime', line 61: type 'features/utime' is not valid or not found on this machine
[2019-04-29 09:40:24.271482] E [MSGID: 101019] [graph.y:320:volume_end] 0-parser: "type" not specified for volume k8s-glusterfs-utime
[2019-04-29 09:40:24.271669] E [MSGID: 100026] [glusterfsd.c:2307:glusterfs_process_volfp] 0-: failed to construct the graph
[2019-04-29 09:40:24.272034] E [graph.c:982:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x12e) [0x56378367338e] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x148) [0x56378366ddc8] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7f676cb0468b] ) 0-graph: invalid argument: graph [Invalid argument]
[2019-04-29 09:40:32.670859] E [MSGID: 101002] [graph.y:212:volume_type] 0-parser: Volume 'k8s-glusterfs-utime', line 61: type 'features/utime' is not valid or not found on this machine
[2019-04-29 09:40:32.670935] E [MSGID: 101019] [graph.y:320:volume_end] 0-parser: "type" not specified for volume k8s-glusterfs-utime
[2019-04-29 09:40:32.671118] E [MSGID: 100026] [glusterfsd.c:2307:glusterfs_process_volfp] 0-: failed to construct the graph
[2019-04-29 09:40:32.671504] E [graph.c:982:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x12e) [0x55d7af7e738e] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x148) [0x55d7af7e1dc8] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7f33da2a568b] ) 0-graph: invalid argument: graph [Invalid argument]
[2019-04-29 09:40:48.870810] E [MSGID: 101002] [graph.y:212:volume_type] 0-parser: Volume 'k8s-glusterfs-utime', line 61: type 'features/utime' is not valid or not found on this machine
[2019-04-29 09:40:48.870865] E [MSGID: 101019] [graph.y:320:volume_end] 0-parser: "type" not specified for volume k8s-glusterfs-utime
[2019-04-29 09:40:48.871021] E [MSGID: 100026] [glusterfsd.c:2307:glusterfs_process_volfp] 0-: failed to construct the graph
[2019-04-29 09:40:48.871325] E [graph.c:982:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x12e) [0x55b23825838e] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x148) [0x55b238252dc8] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7fcc6bb9668b] ) 0-graph: invalid argument: graph [Invalid argument]
[2019-04-29 09:41:21.069815] E [MSGID: 101002] [graph.y:212:volume_type] 0-parser: Volume 'k8s-glusterfs-utime', line 61: type 'features/utime' is not valid or not found on this machine
[2019-04-29 09:41:21.069878] E [MSGID: 101019] [graph.y:320:volume_end] 0-parser: "type" not specified for volume k8s-glusterfs-utime
[2019-04-29 09:41:21.070033] E [MSGID: 100026] [glusterfsd.c:2307:glusterfs_process_volfp] 0-: failed to construct the graph
[2019-04-29 09:41:21.070404] E [graph.c:982:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x12e) [0x55c07e7d838e] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x148) [0x55c07e7d2dc8] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7fe065eee68b] ) 0-graph: invalid argument: graph [Invalid argument]

发现他一直在提示这一句 Volume 'k8s-glusterfs-utime', line 61: type 'features/utime' is not valid or not found on this machine
造成这个原因有如下几种可能:
1,glusterfs-server和glusterfs-client安装的版本不对应(解决办法,升级版本使版本一致)
2,glusterfs-server和glusterfs-clinet时间不同步,(做ntp时间服务器,或者关闭校验使用命令 gluster volume set $VolumeName ctime off)