问题描述

在原来的老集群笔者为K8S 1.14.3的环境中测试nfs-csi的功能，发现动态创建pv，是可以跟pvc进行绑定的，但是在pod attach的时候发生报错，describe pod发现报错信息如下：

Feb 16 14:49:36 slave7 kubelet[2523120]: I0216 14:49:36.658610 2523120 reconciler.go:227] operationExecutor.AttachVolume started for volume "pvc-dd48bec6-8ed9-11ec-9a69-fa163e25388c" (UniqueName: "kubernetes.io/csi/nfs.csi.inspur.com^192.168.1.201/pvc-dd48bec6-8ed9-11ec-9a69-fa163e25388c") pod "nexus-5bd6475fc5-zgh9q" (UID: "bfd510da-8ef3-11ec-9a69-fa163e25388c")
Feb 16 14:49:36 slave7 kubelet[2523120]: E0216 14:49:36.660467 2523120 csi_attacher.go:93] kubernetes.io/csi: attacher.Attach failed: volumeattachments.storage.k8s.io is forbidden: User "system:node:slave7" cannot create resource "volumeattachments" in API group "storage.k8s.io" at the cluster scope: can only get individual resources of this type
Feb 16 14:49:36 slave7 kubelet[2523120]: E0216 14:49:36.660524 2523120 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/csi/nfs.csi.inspur.com^192.168.1.201/pvc-dd48bec6-8ed9-11ec-9a69-fa163e25388c\"" failed. No retries permitted until 2022-02-16 14:51:38.660499769 +0800 CST m=+3307.695433396 (durationBeforeRetry 2m2s). Error: "AttachVolume.Attach failed for volume \"pvc-dd48bec6-8ed9-11ec-9a69-fa163e25388c\" (UniqueName: \"kubernetes.io/csi/nfs.csi.inspur.com^192.168.1.201/pvc-dd48bec6-8ed9-11ec-9a69-fa163e25388c\") from node \"slave7\" : volumeattachments.storage.k8s.io is forbidden: User \"system:node:slave7\" cannot create resource \"volumeattachments\" in API group \"storage.k8s.io\" at the cluster scope: can only get individual resources of this type"
Feb 16 14:49:49 slave7 kubelet[2523120]: W0216 14:49:49.223559 2523120 kubelet_pods.go:849] Unable to retrieve pull secret harbor/service-registry for harbor/harbor-redis-redis4-ha-server-1 due to secret "service-registry" not found.  The image pull may not succeed.
Feb 16 14:50:05 slave7 kubelet[2523120]: E0216 14:50:05.223754 2523120 kubelet.go:1665] Unable to mount volumes for pod "nexus-5bd6475fc5-zgh9q_cicd(bfd510da-8ef3-11ec-9a69-fa163e25388c)": timeout expired waiting for volumes to attach or mount for pod "cicd"/"nexus-5bd6475fc5-zgh9q". list of unmounted volumes=[nexus-data nexus-backup]. list of unattached volumes=[nexus-data nexus-backup default-token-nh7xp]; skipping pod
Feb 16 14:50:05 slave7 kubelet[2523120]: E0216 14:50:05.223799 2523120 pod_workers.go:190] Error syncing pod bfd510da-8ef3-11ec-9a69-fa163e25388c ("nexus-5bd6475fc5-zgh9q_cicd(bfd510da-8ef3-11ec-9a69-fa163e25388c)"), skipping: timeout expired waiting for volumes to attach or mount for pod "cicd"/"nexus-5bd6475fc5-zgh9q". list of unmounted volumes=[nexus-data nexus-backup]. list of unattached volumes=[nexus-data nexus-backup default-token-nh7xp]

具体错误为这一句：

kubernetes.io/csi: attacher.Attach failed: volumeattachments.storage.k8s.io is forbidden: User "system:node:slave7" cannot create resource "volumeattachments" in API group "storage.k8s.io" at the cluster scope: can only get individual resources of this type

排查过程

1. 排查日志发现根本没有走到nfs-csi插件
2. 排查kubelet日志发现报错跟上面一致，分析一下日志感觉应该是权限问题
3. 排查cluseterrole 发现有一个system:node的clusterrole
4. kubectl get ClusterRole system:node -o yaml 结果太长，只看相关的：

...
- apiGroups:
  - storage.k8s.io
  resources:
  - volumeattachments
  verbs:
  - get
 ...

好了，这就找到原因了，system:node 对于 storage.k8s.io group的volumeattachments没有create权限，导致了上面的报错

解决办法

解决办法也很简单，改一下这个system:node的clusterrole就可以了

kubectl edit ClusterRole system:node

- apiGroups:
  - storage.k8s.io
  resources:
  - volumeattachments
  verbs:
  - create
  - delete
  - get
  - patch
  - update
- apiGroups:

然后就搞定了

后续

后面排查了K8S 1.20集群，发现这个clusterrole已经改了，所以应该是1.14版本没有适配的问题