部署方式：

csi-provisioner-cfsplugin

[kube-system/statefulset]

1.csi-provisioner

社区 sidecar，负责监听 pvc 资源，并通知 cfs 进行文件系统的创建与删除。
2.csi-cfs

自研 csi 插件，负责接收 csi-provisioner 请求，并调用对应腾讯云文件系统服务中文件系统的创建删除接口。
3.tcfs

自研 operator 插件，针对共享类型的sc。

csi-nodeplugin-cfsplugin

[kube-system/daemonset]

1.csi-node-driver-registrar

社区 sidecar，负责向 kubelet 注册对应 csi 插件。
2.csi-cfs

自研 csi 插件，负责接收 kubelet 请求，进行文件系统的 mount 与 umount（共享存储也由该插件进行文件系统的 mount 与 umount）。

csi-attacher-cfsplugin

[kube-system/statefulset]

集群版本小于1.14才安装

1.csi-attacher

通过GetControllerCapabilities接口，检测到CSI不支持attach.dettach，

启动trivialHandler。

trivialHandler，会自动将VA更新为attached.

2.csi-cfs

GetControllerCapabilities接口

1.14以上版本，CSIDriver设置了不需要attach。

apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  annotations:
    meta.helm.sh/release-name: cfs
    meta.helm.sh/release-namespace: kube-system
  name: com.tencent.cloud.csi.cfs
spec:
  attachRequired: false
  fsGroupPolicy: File
  podInfoOnMount: false
  requiresRepublish: false
  storageCapacity: false
  volumeLifecycleModes:
  - Persistent

TCFS

tcfs的CRD资源

 

 

1.非共享类型的CFS

1.1 创建SC和PVC

SC


apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cfs-normal
parameters:
  pgroupid: pgroup-lrquouvl // 权限组(针对网络访问)
  storagetype: SD // CFS实例的类型：标准/性能
  subnetid: subnet-rtwb42lu // 子网
  vers: "3" // NFS协议版本
  vpcid: vpc-oilua6pt // VPC网络
  zone: ap-guangzhou-3 // 可用区
provisioner: com.tencent.cloud.csi.cfs
reclaimPolicy: Delete
volumeBindingMode: Immediate

PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: com.tencent.cloud.csi.cfs
    volume.kubernetes.io/storage-provisioner: com.tencent.cloud.csi.cfs
  finalizers:
  - kubernetes.io/pvc-protection
  name: cfs-normal
  namespace: default
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  storageClassName: cfs-normal
  volumeMode: Filesystem
  volumeName: pvc-a11e120e-130c-4aff-9901-17b7b91b3f1d
status:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 10Gi
  phase: Bound
 

1.2 自动创建PV
 

1）csi-provisioner-cfsplugin/csi-provisioner
 

调用createvolume接口，创建CFS。
 

日志：
 

I0221 09:46:37.456573       1 controller.go:1317] provision "default/cfs-normal" class "cfs-normal": started
I0221 09:46:37.457251       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"cfs-normal", UID:"a11e120e-130c-4aff-9901-17b7b91b3f1d", APIVersion:"v1", ResourceVersion:"424215490", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/cfs-normal"
I0221 09:46:44.915159       1 controller.go:655] create volume rep: {CapacityBytes:10737418240 VolumeId:cfs-3esjn2lz VolumeContext:map[fsid:bz8eisu7 host:10.0.33.133 pgroupid:pgroup-lrquouvl storagetype:SD subnetid:subnet-rtwb42lu vers:3 vpcid:vpc-oilua6pt zone:ap-guangzhou-3] ContentSource:<nil> AccessibleTopology:[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0221 09:46:44.915242       1 controller.go:737] successfully created PV pvc-a11e120e-130c-4aff-9901-17b7b91b3f1d for PVC cfs-normal and csi volume name cfs-3esjn2lz
I0221 09:46:44.915257       1 controller.go:1420] provision "default/cfs-normal" class "cfs-normal": volume "pvc-a11e120e-130c-4aff-9901-17b7b91b3f1d" provisioned
I0221 09:46:44.915292       1 controller.go:1437] provision "default/cfs-normal" class "cfs-normal": succeeded
E0221 09:46:44.932235       1 controller.go:1443] couldn't create key for object pvc-a11e120e-130c-4aff-9901-17b7b91b3f1d: object has no meta: object does not implement the Object interfaces
I0221 09:46:44.932419       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"cfs-normal", UID:"a11e120e-130c-4aff-9901-17b7b91b3f1d", APIVersion:"v1", ResourceVersion:"424215490", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-a11e120e-130c-4aff-9901-17b7b91b3f1d
 

2）csi-provisioner-cfsplugin/csi-tencentcloud-cfs
 

调用CFS接口，
 

I0221 09:46:37.457486       1 controller.go:42] CreateVolume CreateVolumeRequest is name:"pvc-a11e120e-130c-4aff-9901-17b7b91b3f1d" capacity_range:<required_bytes:10737418240 > volume_capabilities:<mount:<> access_mode:<mode:MULTI_NODE_MULTI_WRITER > > parameters:<key:"pgroupid" value:"pgroup-lrquouvl" > parameters:<key:"storagetype" value:"SD" > parameters:<key:"subnetid" value:"subnet-rtwb42lu" > parameters:<key:"vers" value:"3" > parameters:<key:"vpcid" value:"vpc-oilua6pt" > parameters:<key:"zone" value:"ap-guangzhou-3" > :
I0221 09:46:37.457599       1 controller.go:58] req.name is :  pvc-a11e120e-130c-4aff-9901-17b7b91b3f1d   
I0221 09:46:37.457616       1 secret_util.go:20] Get secretID or secretKey from env failed, will use cloud norm!
 

1.3 创建POD绑定PV
 

1）kubelet
 

// attach，不支持，假设已经attached
2月 21 17:58:43 VM-33-198-centos kubelet[4762]: I0221 17:58:43.484139    4762 reconciler.go:342] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-a11e120e-130c-4aff-9901-17b7b91b3f1d\" (UniqueName: \"kubernetes.io/csi/com.tencent.cloud.csi.cfs^cfs-3esjn2lz\") pod \"tao-86c948ff77-m8t58\" (UID: \"4ce98f8e-1ff8-4d78-b61c-739997ff026e\") " pod="default/tao-86c948ff77-m8t58"
2月 21 17:58:43 VM-33-198-centos kubelet[4762]: I0221 17:58:43.484214    4762 reconciler.go:342] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmb6v\" (UniqueName: \"kubernetes.io/projected/4ce98f8e-1ff8-4d78-b61c-739997ff026e-kube-api-access-cmb6v\") pod \"tao-86c948ff77-m8t58\" (UID: \"4ce98f8e-1ff8-4d78-b61c-739997ff026e\") " pod="default/tao-86c948ff77-m8t58"
​
2月 21 17:58:43 VM-33-198-centos kubelet[4762]: I0221 17:58:43.585483    4762 reconciler.go:254] "operationExecutor.MountVolume started for volume \"pvc-a11e120e-130c-4aff-9901-17b7b91b3f1d\" (UniqueName: \"kubernetes.io/csi/com.tencent.cloud.csi.cfs^cfs-3esjn2lz\") pod \"tao-86c948ff77-m8t58\" (UID: \"4ce98f8e-1ff8-4d78-b61c-739997ff026e\") " pod="default/tao-86c948ff77-m8t58"
// 全局mount，不支持跳过
2月 21 17:58:43 VM-33-198-centos kubelet[4762]: I0221 17:58:43.601243    4762 csi_attacher.go:358] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice...
2月 21 17:58:43 VM-33-198-centos kubelet[4762]: I0221 17:58:43.601309    4762 operation_generator.go:658] "MountVolume.MountDevice succeeded for volume \"pvc-a11e120e-130c-4aff-9901-17b7b91b3f1d\" (UniqueName: \"kubernetes.io/csi/com.tencent.cloud.csi.cfs^cfs-3esjn2lz\") pod \"tao-86c948ff77-m8t58\" (UID: \"4ce98f8e-1ff8-4d78-b61c-739997ff026e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/com.tencent.cloud.csi.cfs/bc6b83b6ab76c3374861a4a35a20a7972c79c506e5b47d412f46a1d63e6b9516/globalmount\"" pod="default/tao-86c948ff77-m8t58"
// mount
2月 21 17:58:43 VM-33-198-centos kubelet[4762]: I0221 17:58:43.615495    4762 operation_generator.go:703] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmb6v\" (UniqueName: \"kubernetes.io/projected/4ce98f8e-1ff8-4d78-b61c-739997ff026e-kube-api-access-cmb6v\") pod \"tao-86c948ff77-m8t58\" (UID: \"4ce98f8e-1ff8-4d78-b61c-739997ff026e\") " pod="default/tao-86c948ff77-m8t58"
2月 21 17:58:43 VM-33-198-centos kubelet[4762]: I0221 17:58:43.656027    4762 operation_generator.go:703] "MountVolume.SetUp succeeded for volume \"pvc-a11e120e-130c-4aff-9901-17b7b91b3f1d\" (UniqueName: \"kubernetes.io/csi/com.tencent.cloud.csi.cfs^cfs-3esjn2lz\") pod \"tao-86c948ff77-m8t58\" (UID: \"4ce98f8e-1ff8-4d78-b61c-739997ff026e\") " pod="default/tao-86c948ff77-m8t58"
 

 

2）csi-nodeplugin-cfsplugin/csi-cfs
 

// mount
I0221 09:58:43.604953       1 node.go:56] NodePublishVolume NodePublishVolumeRequest is: volume_id:"cfs-3esjn2lz" target_path:"/var/lib/kubelet/pods/4ce98f8e-1ff8-4d78-b61c-739997ff026e/volumes/kubernetes.io~csi/pvc-a11e120e-130c-4aff-9901-17b7b91b3f1d/mount" volume_capability:<mount:<> access_mode:<mode:MULTI_NODE_MULTI_WRITER > > volume_context:<key:"fsid" value:"bz8eisu7" > volume_context:<key:"host" value:"10.0.33.133" > volume_context:<key:"pgroupid" value:"pgroup-lrquouvl" > volume_context:<key:"storage.kubernetes.io/csiProvisionerIdentity" value:"1676950325113-8081-com.tencent.cloud.csi.cfs" > volume_context:<key:"storagetype" value:"SD" > volume_context:<key:"subnetid" value:"subnet-rtwb42lu" > volume_context:<key:"vers" value:"3" > volume_context:<key:"vpcid" value:"vpc-oilua6pt" > volume_context:<key:"zone" value:"ap-guangzhou-3" > 
I0221 09:58:43.605662       1 node.go:150] CFS server 10.0.33.133:/bz8eisu7/ mount option is: [vers=3 noresvport nolock,proto=tcp]
I0221 09:59:03.388362       1 node.go:223] Enabling node service capability: GET_VOLUME_STATS
I0221 09:59:03.388377       1 node.go:223] Enabling node service capability: UNKNOWN
// 
I0221 09:59:03.391793       1 node.go:240] NodeGetVolumeStats is: volume_id:"cfs-3esjn2lz" volume_path:"/var/lib/kubelet/pods/4ce98f8e-1ff8-4d78-b61c-739997ff026e/volumes/kubernetes.io~csi/pvc-a11e120e-130c-4aff-9901-17b7b91b3f1d/mount" 
 

 

2.共享类型的CFS
 

10.0.1.128:/m002rxiu/default-cfs-share-pvc-58c574ae-a636-4296-aed9-ff1eec39de56    /var/lib/kubelet/pods/bc08cf37-1e19-4d6c-b350-98e4b5cb86b6/volumes/kubernetes.io~csi/pvc-58c574ae-a636-4296-aed9-ff1eec39de56/mount
 

 

在SC/PVC中，driver为“com.tencent.cloud.csi.tcfs.<SC的名字>”
 

在PV的spec.csi中，driver为“com.tencent.cloud.csi.cfs”。
 

2.1 创建SC和PVC
 

SC
 

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cfs-share
parameters:
  pgroupid: pgroup-lrquouvl // 权限组(针对网络访问)
  storagetype: SD // CFS实例的类型：标准/性能
  subdir-share: "true" // 共享模式
  vers: "3" // NFS协议版本
  subnetid: subnet-rtwb42lu // 子网
  vpcid: vpc-oilua6pt // VPC网络
  zone: ap-guangzhou-3 // 可用区
provisioner: com.tencent.cloud.csi.tcfs.cfs-share // 以SC的名字，作为CSI类型
reclaimPolicy: Retain
volumeBindingMode: Immediate
 

PVC
 



apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: com.tencent.cloud.csi.tcfs.cfs-share
    volume.kubernetes.io/storage-provisioner: com.tencent.cloud.csi.tcfs.cfs-share
  finalizers:
  - kubernetes.io/pvc-protection
  name: cfs-share-1
  namespace: default
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  storageClassName: cfs-share
  volumeMode: Filesystem
  volumeName: pvc-651c987e-70c5-4542-93c1-2ba8184ad862
status:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 10Gi
  phase: Bound
 

2.2 为SC创建tcfs和deployment
 

1）创建的内容：
 

deoloyment
 

 

 
 

 

TCFS
 

apiVersion: tcfsoperator.k8s.io/v1alpha1
kind: Tcfs
metadata:
  name: cfs-share
  namespace: kube-system
spec:
  pgroupid: pgroup-lrquouvl // 权限组
  provisionername: com.tencent.cloud.csi.tcfs.cfs-share // CSI的名字
  reclaimpolicy: Retain
  storagetype: SD // CFS实例类型
  subnetid: subnet-rtwb42lu
  vpcid: vpc-oilua6pt
  zone: ap-guangzhou-3
status:
  cfsfilesystemid: wmyio3c8 // FSID
  cfsserver: 10.0.33.130 // CFS实例的IP
 

 

2）csi-provisioner-cfsplugin中的TCFS容器
 

storageclass_controller 
1）处理PVC
 
根据PVC，拿到对应的SC。
 
如果SC为共享模式，创建对于tcfs对象。
 
2）处理SC
 
storageClass被删除,删除对应的tcf对象
 
tcfs_controller 
监听CS对象。为SC对应的tcfs对象创建CFS实例，更新status，创建deploy。
 
SC和TCFS的命名空间的名字相同。
 
----------
 
步骤1:查询SC对应的TCFS的实际情况
 
1.查询TCFS是否存在
 
2.使用3种方法: 尝试获取TCFS，关联的cfs信息
 
1）根据tcfs.status的CfsServer和CfsFileSystemID字段
 
2）根据tcfs.Spec.FromExistCfs中记录的CFS实例ID
 
3）使用固定的CFS名字“<集群ID>_sharedCFS-<SC 名字>”，查询CFS实例
 
3.查询tcfs关联的deployment
 
步骤2:生成期望的TCFS状态
 
1）生成tcfs对应deploy的YAML文件。
 
步骤3:同步期望状态和实际状态
 
1）创建CFS实例【不删除CFS实例】
 
直接调用CFS接口创建。
 
2）更新或创建或删除deploy
 
 

 

2.3 SC对于的deploy，为PVC创建PV
 

针对共享类型的storgeclass，会创建下面的deployment。
 

com.tencent.cloud.csi.tcfs.<storgeclass的名字>
 

1）仓库
 

开源组件：GitHub - kubernetes-sigs/nfs-subdir-external-provisioner: Dynamic sub-dir volume provisioner on a remote NFS server.
 

2）启动参数
 

 

 
 

环境变量PROVISIONER_NAME：设置了要监听的CSI类型。
 

3) volume
 

 

 
 

 

4）作用：
 

NFS subdir external provisioner是一个自动配置器，使用现存已经配置好的NF服务器，来支持为PVC动态创建PV。
 

实现了Provisioner接口的两个方法：
 

Provision 
1)在CFS下，创建子路径：
 
<PVC ns>-<PVC name>-<PV name>/自定于的路径模式
 
自定于的路径模式：在SC的"pathPattern"指定，可以为空
 
2)生成PV
 
 
 
 
Delete 
1.根据sc的"onDelete"
 
判断是否删除/保留路径
 
2.根据sc的"archiveOnDelete"
 
T：将自路径重命名为“archived-<原路径>”
 
 

日志：
 

I0221 03:38:37.616545       1 controller.go:1317] provision "default/cfs-share-1" class "cfs-share": started
I0221 03:38:37.629793       1 event.go:278] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"cfs-share-1", UID:"651c987e-70c5-4542-93c1-2ba8184ad862", APIVersion:"v1", ResourceVersion:"419591929", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/cfs-share-1"
I0221 03:38:37.633497       1 controller.go:1420] provision "default/cfs-share-1" class "cfs-share": volume "pvc-651c987e-70c5-4542-93c1-2ba8184ad862" provisioned
I0221 03:38:37.633541       1 controller.go:1437] provision "default/cfs-share-1" class "cfs-share": succeeded
I0221 03:38:37.633552       1 volume_store.go:212] Trying to save persistentvolume "pvc-651c987e-70c5-4542-93c1-2ba8184ad862"
I0221 03:38:37.649191       1 volume_store.go:219] persistentvolume "pvc-651c987e-70c5-4542-93c1-2ba8184ad862" saved
I0221 03:38:37.649426       1 event.go:278] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"cfs-share-1", UID:"651c987e-70c5-4542-93c1-2ba8184ad862", APIVersion:"v1", ResourceVersion:"419591929", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-651c987e-70c5-4542-93c1-2ba8184ad862
 

2.4 创建POD，PVmount流程
 

在PV的spec.csi中，driver为“com.tencent.cloud.csi.cfs”。
 

与非共享类型的流程一致。
 
  

        # kubernetes
      
  
 
 Logo 
K8S/Kubernetes
 
K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容
 
 
  
更多推荐
  
 
【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美
 
作者 |  韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号​前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。​长夜有穷，真水无香。领略过了 K8s“身在江
 
avatar K8S/Kubernetes
 
 
如何基于 K8s 构建下一代 DevOps 平台？
 
作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub
 
avatar K8S/Kubernetes
 
 
k8s 火了！
 
2020，上云之年，产品云端化成为一种趋势。在一线城市，很多公司都已经构建了自己的私有云环境，比如阿里云、网易云、华为云等。而Kubernetes 作为基于容器编排领域的王者，具备扩展...
 
avatar K8S/Kubernetes