引言：

• 当容器部署涉及到多个节点服务器，用docker、docker-compose 来部署就不是那么方便了，不能统一控制，不方便伸缩，配置统一管理、版本控制；当我们业务涉及到集群容器部署时，就考虑K8s、K3s 来实现。

k8s 架构组成：

• （一）Master ：apiserver、scheduler、controller-manager 以及 ETCD
• （多）Node ： kubelet、kube-proxy、容器引擎（containerd、docker）

Kube-apiserver：

• 用于暴露 kubernetes API，任何资源请求或调用操作都是通过 kube-apiserver 提供的接口进行。以 HTTP Restful API 提供接口服务，所有对象资源的增删改查和监听操作都交给 API Server 处理后再提交给 Etcd 存储。
  可以理解成 API Server 是 K8S 的请求入口服务。API Server 负责接收 K8S 所有请求（来自 UI 界面或者 CLI 命令行工具），然后根据用户的具体请求，去通知其他组件干活。可以说 API Server 是 K8S 集群架构的大脑。

Kube-controller-manager：

• 运行管理控制，是 K8S 集群中处理常规任务的后台线程，是 K8S 集群里所有资源对象的自动化控制中心。
  在 K8S 集群中，一个资源对应一个控制器，而 Controller manager 就是负责管理这些控制器的。由一系列控制器组成，通过 API Server 监控整个集群的状态，并确保集群处于预期的工作状态，比如当某个 Node 意外宕机时，Controller Manager 会及时发现并执行自动化修复流程，确保集群始终处于预期的工作状态。

kube-scheduler：

• 负责资源调度的进程，根据调度算法为新创建的 Pod 选择一份合适的 Node 节点。可以理解成 K8S 所有 Node 节点的调度器。当用于要部署服务时，Scheduler 会根据调度算法选择最合适的 Node 节点来部署 Pod

etcd 配置存储中心：

• 8S 的存储服务。etcd 分布式键值存储系统，存储了 K8S 的关键配置和用户配置，K8S 中仅 API Server 才具备读写权限，其他组件必须通过 API Server 的接口才能读写数据

Kubelet ：

• Node 节点的监视器，以及与 Master 节点的通讯器。Kubelet 是 Master 节点安插在 Node 节点上的 “眼线”，它会定时向 API Server 汇报自己 Node 节点上运行的服务的状态，并接受来自 Master 节点的指示采取调整措施。
  从 Master 节点获取自己节点上 Pod 的期望状态（比如运行什么容器、运行的副本数量、网络或者存储如何配置等），直接跟容器引擎交互实现容器的生命周期管理，如果自己节点上 Pod 的状态与期望状态不一致，则调用对应的容器平台接口（即 docker 的接口）达到这个状态。管理镜像和容器的清理工作，保证节点上镜像不会占满磁盘空间，退出的容器不会占用太多资源。

Kube-Proxy：

• 在每个 Node 节点上实现 Pod 网络代理，是 Kuberbetes Service 资源的载体，负责维护网络规划和四层负载均衡工作。负责写入规则至 iptables、ipvs 实现服务映射访问的。
  Kube-Proxy 本身不是直接给 Pod 提供网络，Pod 的网络是由 Kubelet 提供的，Kube-Proxy 实际上维护的是虚拟的 Pod 集群网络。
  Kube-apiserver 通过监控 Kube-Proxy 进行对 Kubernets Service 的更新和端点的维护。
  Kube-Proxy 是 K8S 集群内部的负载均衡器。它是一个分布式代理服务器，在 K8S 的每个节点上都会运行一个 Kube-proxy。

pod：

• 1、Pod 是 Kuberntes 创建或部署的最小/最简单的基本单位，一个 Pod 代表集群上正在运行的一个进程。可以把 Pod 理解成豌豆荚，而同一 Pod 内的每个容器是一个个豌豆。
  2、一个 Pod 由一个或多个容器组成，Pod 中容器共享网络、存储和计算资源，在同一台 Docker 主机上运行。
  3、一个 Pod 里可以运行多个容器，又叫边车（SideCara）模式。而在生产环境中一般都是单个容器或者具有强关联互补的多个容器组成一个 Pod。
  4、同一个 Pod 之间的容器可以通过 localhost 互相访问，并且可以挂载 Pod 内的所有的数据卷；但是不同的 Pod 之间的容器不能用 localhost 访问，也不能挂载其他 Pod 的数据卷。

常用命令：

#获取默认 命名空间pod列表
kubectl get pod
#获取 pod详情包含部署节点、容器ID
kubectl get pod -o wide
#删除指定pod -默认命名空间
kubectl delete pod test-pod
#查询 pod进程描述 xx -pod名称
kubectl describe pod xx
#查询 xx pod日志信息
kubectl logs xx
#查询 xx pod日志信息 持续输出
kubectl logs xx -f
#指定命名空间，查看pod 日志
kubectl logs device-info-868fddfc54-p2cd7 -f -n xiaoshu
# 进入某 xx pod容器内部
kubectl exec -it xx -- bash
#指定命名空间，进去容器内部
kubectl exec -it device-info-868fddfc54-p2cd7 -n xiaoshu -- bash

# 部署yml到指定的命名空间
kubectl apply -f app.yml --namespace testapp
# 获取pod列表 指定命令空间
kubectl get pod -n kube-system
# pod 零时端口映射
kubectl port-forward xx 8080:8080
# 获取默认空间 副本集列表
kubectl get deployment
# 获取默认空间 有状态应用集列表
kubectl get StatefulSet
# 删除 test-k8s 名称副本集
kubectl delete deployment test-k8s
# 调整 xx 副本数量
kubectl scale deployment xx --replicas=10
# 查询 xx 副本集，部署历史
kubectl rollout history deployment xx
# 回退上个版本 
ubectl rollout undo deployment test-k8s
# 回退指定版本 
kubectl rollout undo deployment test-k8s --to-revision=2 

# 查看全部
kubectl get all
# 重新部署
kubectl rollout restart deployment test-k8s
# 命令修改镜像，--record 表示把这个命令记录到操作历史中
kubectl set image deployment test-k8s test-k8s=ccr.ccs.tencentyun.com/k8s-tutorial/test-k8s:v2-with-error --record
# 暂停运行，暂停后，对 deployment 的修改不会立刻生效，恢复后才应用设置
kubectl rollout pause deployment test-k8s
# 恢复
kubectl rollout resume deployment test-k8s
# 输出到文件
kubectl get deployment test-k8s -o yaml >> app2.yaml
# 删除全部资源
kubectl delete all --all
kubectl delete all --all -n xiaoshu

#  挂载数据集相关
#  先删除pod再删除pvc最后删除pv
kubectl get pv -n xxx
kubectl get pvc -n xxx

# 简写：cm
# 应用
kubectl apply -f configmap.yaml
# 查看
kubectl get configmap mongo-config -o yaml
#挂载配置文件
kubectl create configmap [configmap名称] --from-file=[文件]

# crictl -k8s中CRI（容器运行时接口）的客户端
crictl images

# containerd
ctr images import xx.tar
# -- 指定导入：
ctr --namespace k8s.io image import xx.tar 

ctr images list|grep xx

java client 对接 k8s api-server

• 配置admin token
• 引入io.kubernetes - client-java
• 搭建私有仓库 harbor

<dependency>
   <groupId>io.kubernetes</groupId>
    <artifactId>client-java</artifactId>
    <version>17.0.0</version>
</dependency>

初始化 client

@Slf4j
@Component
public class K8sApiConfig {

    @Resource
    private VirtualProperties virtualProperties;

    @Bean
    public ApiClient k8sClient(){
        ApiClient client = new ClientBuilder().
                setBasePath(virtualProperties.getK8sUrl()).setVerifyingSsl(false).
                setAuthentication(new AccessTokenAuthentication(virtualProperties.getK8sToken())).build();
        Configuration.setDefaultApiClient(client);
        return client;
    }

}

virtual:
  docker-url: tcp://192.168.2.207:8088
  k8s-url: https://192.168.2.207:6443
  k8s-token: xx
  server-url: 192.168.2.207
  server-name: root
  server-pwd: 123456
  tar-path: /opt/tar/
  harbor-url: core.harbor.domain
  harbor-uname: admin
  harbor-pwd: Harbor12345

配置 admin-token yml

#部署 admin-token
kubectl apply -f k8s-admin.yaml
#查询 dashboard-admin开头secret
kubectl get secret -n kube-system|grep dashboard-admin
#查询 token
kubectl describe secret dashboard-admin-token-xx -n kube-system

apiVersion: v1
kind: ServiceAccount
metadata:
  name: dashboard-admin
  namespace: kube-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: dashboard-admin
subjects:
  - kind: ServiceAccount
    name: dashboard-admin
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

api相关代码

• vo对象封装太多了，就不列了；这里列个 service 给个参考

public interface K8sService {

    /**
     * 集群节点信息获取
     * @return List<ClusterInfoVo>
     */
    List<ClusterInfoVo> clusterInfo();

    /**
     * 导入镜像包
     * @param file tar文件
     * @return Boolead
     */
    R<String> importTar(MultipartFile file);

    /**
     * k8s 镜像列表查询
     * @param name 镜像名称
     * @return
     */
    List<String> k8sImages(String name);

    /**
     * 命名空间列表
     * @return List<NameSpaceVo>
     */
    List<NameSpaceVo> nameSpaces();

    /**
     * 部署副本集
     * @param deploymentAddVo 副本集参数
     * @return Boolean
     */
    Boolean applyDeployment(DeploymentAddVo deploymentAddVo);

    /**
     * 部署状态应用副本集
     * @param statefulSetAddVo 参数
     * @return Boolean
     */
    Boolean applyStatefulSet(StatefulSetAddVo statefulSetAddVo);

    /**
     * 副本集列表
     * @param nameSpace 命令空间
     * @return List<DeploymentVo>
     */
    List<DeploymentVo> deployments(String nameSpace);

    /**
     * 挂载数据卷查询
     * @param nameSpace 命名空间
     */
    List<VolumesPvVo> volumes(String nameSpace);

    /**
     * 数据集申请单查询
     */
    List<VolumesPvClaimVo> volumesClaim(String nameSpace);

    /**
     * 添加-挂载数据卷申请单
     * @param volumesAddVo 数据集参数
     */
    Boolean addVolumesClaim(VolumesAddVo volumesAddVo);

    /**
     * 添加-挂载数据卷
     * @param volumesAddVo 数据集参数
     */
    Boolean addVolumes(VolumesAddVo volumesAddVo);

    /**
     * 服务列表
     * @param nameSpace 命令空间
     * @return List<ServiceAddVo>
     */
    List<ServiceAddVo> services(String nameSpace);

    /**
     * 服务部署
     * @param serviceAddVo 服务创建参数
     * @return Boolean
     */
    Boolean applyService(ServiceAddVo serviceAddVo);

    /**
     * 删除服务
     * @param deleteK8sVo
     * @return
     */
    Boolean deleteService(DeleteK8sVo deleteK8sVo);

    /**
     * 删除副本集
     * @param deleteK8sVo
     * @return
     */
    Boolean deleteDeployment(DeleteK8sVo deleteK8sVo);

    /**
     * 创建configMap
     * @return Boolean
     */
    Boolean addConfigMap( ConfigMapVo configMapVo);
}

@Slf4j
@Service
public class K8sServiceImpl implements K8sService {

    @Resource
    private VirtualProperties virtualProperties;

    @Override
    public List<ClusterInfoVo> clusterInfo() {
        CoreV1Api api = new CoreV1Api();
        GenericKubernetesApi<NodeMetrics, NodeMetricsList> nodeMetricsListGenericKubernetesApi =  new GenericKubernetesApi<>(
                NodeMetrics.class,
                NodeMetricsList.class,
                "metrics.k8s.io",
                "v1beta1",
                "nodes",
                api.getApiClient());
        NodeMetricsList nodeMetricsList = nodeMetricsListGenericKubernetesApi.list().getObject();
        HashMap<String, Map<String, String>> usages = new HashMap<>();
        for (NodeMetrics item : nodeMetricsList.getItems()) {
            String name = item.getMetadata().getName();
            Map<String, Quantity> usage = item.getUsage();
            Map<String, String> quantity = new HashMap<>();
            for (String s : usage.keySet()) {
                Quantity quantity1 = usage.get(s);
                if (s.equals("cpu")){
                    String cpu = quantity1.toSuffixedString().replace("n","");
                    double v = NumberUtil.div(Long.parseLong(cpu), 1000_000_000, 2);
                    quantity.put(s,String.valueOf(v));
                }else {
                    long curByte = quantity1.getNumber().longValue();
                    quantity.put(s,String.valueOf(curByte));
                }
            }
            usages.put(name,quantity);
        }

        List<ClusterInfoVo> clusterInfoVos = new LinkedList<>();
        try {
            V1NodeList nodeList = api.listNode("true", false, null, null, null, null, null, null, 60, false);
            List<V1Node> items = nodeList.getItems();
            for (V1Node node : items) {
                ClusterInfoVo clusterInfoVo = new ClusterInfoVo();
                V1NodeStatus nodeStatus = node.getStatus();
                //节点镜像列表
                List<V1ContainerImage> images = nodeStatus.getImages();

                List<V1NodeAddress> addresses = nodeStatus.getAddresses();
                //node节点ip地址
                String nodeIp = addresses.get(0).getAddress();
                clusterInfoVo.setClusterId(nodeIp);
                String hostName = addresses.get(1).getAddress();
                clusterInfoVo.setNodeName(hostName);
                //容量信息
                Map<String, Quantity> capacity = nodeStatus.getCapacity();
                capacity.forEach((k,v)->{
                    //cpu 核数
                    if (k.equals("cpu")){
                        BigDecimal cpu = v.getNumber();
                        Long totalCpu = cpu.longValue();
                        clusterInfoVo.setCpuCores(cpu.intValue());
                        if(usages.get(hostName)!=null){
                            String useCpu = usages.get(hostName).get("cpu");
                            clusterInfoVo.setCpuUses(useCpu);
                            double v1 = NumberUtil.div(Double.valueOf(useCpu).doubleValue(), totalCpu.doubleValue(), 2);
                            BigDecimal scale = new BigDecimal(v1).multiply(new BigDecimal(100)).setScale(0, RoundingMode.HALF_UP);
                            clusterInfoVo.setCpuPercent(scale.toString());
                            clusterInfoVo.setOnlineStatus(Boolean.TRUE);
                        }else {
                            clusterInfoVo.setOnlineStatus(Boolean.FALSE);
                        }
                    }
                    //临时存储
                    if (k.equals("ephemeral-storage")){
                        BigDecimal ephemeral = v.getNumber();
                        clusterInfoVo.setDiskSize(LocalFileUtil.fileShow(ephemeral.longValue()));
                    }
                    //内存
                    if (k.equals("memory")){
                        BigDecimal memory = v.getNumber();
                        long totalMemory = memory.longValue();
                        clusterInfoVo.setMemory(LocalFileUtil.fileShow(totalMemory));
                        if (usages.get(hostName)!=null){
                            Long useMemory = Long.parseLong(usages.get(hostName).get("memory"));
                            clusterInfoVo.setUserMemory(LocalFileUtil.fileShow(useMemory));
                            double v1 = NumberUtil.div(useMemory.longValue(), totalMemory, 2);
                            BigDecimal scale = new BigDecimal(v1).multiply(new BigDecimal(100)).setScale(0, RoundingMode.HALF_UP);
                            clusterInfoVo.setMemoryPercent(scale.toString());
                        }
                    }
                    //运行容器数
                    if (k.equals("pod")){
                        BigDecimal pods = v.getNumber();
                        clusterInfoVo.setPods(pods.intValue());
                    }
                });
                V1NodeSystemInfo nodeInfo = nodeStatus.getNodeInfo();
                //处理器
                String architecture = nodeInfo.getArchitecture();
                clusterInfoVo.setArchitecture(architecture);
                //机器编号
                String machineID = nodeInfo.getMachineID();
                clusterInfoVo.setMachineID(machineID);
                //操作系统
                String operatingSystem = nodeInfo.getOperatingSystem();
                clusterInfoVo.setOperatingSystem(operatingSystem);
                //系统镜像
                String osImage = nodeInfo.getOsImage();
                clusterInfoVo.setOsImage(osImage);
                V1ObjectMeta metadata = node.getMetadata();
                OffsetDateTime dateTime = metadata.getCreationTimestamp();
                Date date = Date.from(dateTime.toLocalDateTime().toInstant(ZoneOffset.ofHours(8)));
                clusterInfoVo.setCreateTime(date);
                clusterInfoVos.add(clusterInfoVo);
            }
        } catch (ApiException e) {
            e.printStackTrace();
        }
        return clusterInfoVos;
    }

    @Override
    public R<String> importTar(MultipartFile file) {
        long size = file.getSize();
        if (size==0){
            return R.failed("不能上传空文件");
        }
        long fileName = IdUtil.getSnowflakeNextId();
        String originalFilename = file.getOriginalFilename();
        log.info("originalFilename:"+originalFilename);
        String suffix = FileUtil.getSuffix(originalFilename);
        log.info("suffix:"+suffix);
        if (!"tar".equals(suffix)){
            return R.failed("请上传镜像文件tar包");
        }
        try {
            String tarName = fileName +"."+ suffix;
            File tarFile = new File(virtualProperties.getTarPath() + tarName);
            file.transferTo(tarFile);
            if (tarFile.exists()){
                log.info("tarName:"+tarName);
                List<String> resMsg = ExecUtil.remoteExec("ctr --namespace k8s.io image import "+virtualProperties.getTarPath()+ tarName,
                        virtualProperties.getServerUrl(),
                        virtualProperties.getServerName(), virtualProperties.getServerPwd());
                boolean done = resMsg.stream().allMatch(e -> e.contains("done"));
                if (done){
                    return R.ok("上传成功");
                }else {
                    return R.failed("上传失败");
                }
            }else {
                return R.failed("上传失败");
            }
        } catch (IOException e) {
            e.printStackTrace();
            return R.failed("上传失败");
        }
    }

    @Override
    public List<String> k8sImages(String name) {
        //V1Node status = api.re(node.getMetadata().getName(), "true");
        List<String> resMsg = new LinkedList<>();
        try {
            String cmd="crictl images";
            if (StrUtil.isNotEmpty(name)){
                cmd=cmd+"|grep "+name;
            }
            resMsg = ExecUtil.remoteExec(cmd,virtualProperties.getServerUrl(),
                    virtualProperties.getServerName(), virtualProperties.getServerPwd());
        } catch (IOException e) {
            e.printStackTrace();
        }
        return resMsg;
    }

    @Override
    public List<NameSpaceVo> nameSpaces() {
        List<NameSpaceVo> res = new LinkedList<>();
        CoreV1Api api = new CoreV1Api();
        V1NamespaceList listNamespace = null;
        try {
            listNamespace = api.listNamespace("true", null, null, null, null, null, null, null, null, false);
        } catch (ApiException e) {
            e.printStackTrace();
        }
        if (listNamespace==null){
            return res;
        }
        List<V1Namespace> items = listNamespace.getItems();
        items.forEach(e->{
            NameSpaceVo nameSpaceVo = new NameSpaceVo();
            String status = e.getStatus().getPhase();
            nameSpaceVo.setStatus(status);
            V1ObjectMeta meta = e.getMetadata();
            String name = meta.getName();
            nameSpaceVo.setName(name);
            OffsetDateTime dateTime = meta.getCreationTimestamp();
            Date date = Date.from(dateTime.toLocalDateTime().toInstant(ZoneOffset.ofHours(8)));
            nameSpaceVo.setCreateTime(date);
            res.add(nameSpaceVo);
        });
        return res;
    }

    @Override
    public Boolean applyDeployment(DeploymentAddVo deploymentAddVo) {
        AppsV1Api api = new AppsV1Api();
        V1ObjectMeta meta = new V1ObjectMeta();
        if (StrUtil.isEmpty(deploymentAddVo.getName())){
            log.info("副本集名称不能为空");
            return Boolean.FALSE;
        }
        if (StrUtil.isEmpty(deploymentAddVo.getNameSpace())){
            log.info("需要指定命名空间");
            return Boolean.FALSE;
        }
        meta.setName(deploymentAddVo.getName());
        meta.setNamespace(deploymentAddVo.getNameSpace());
        List<LabelsOrKvVo> labelVo = deploymentAddVo.getLabels();
        if (CollectionUtil.isEmpty(labelVo)){
            log.info("至少需要输入一个标签");
            return Boolean.FALSE;
        }
        Map<String,String> labels =  new HashMap<>(labelVo.size());
        for (LabelsOrKvVo labelsOrKvVo : labelVo) {
            labels.put(labelsOrKvVo.getKey(), labelsOrKvVo.getValue());
        }
        meta.setLabels(labels);
        V1Deployment v1Deployment = new V1Deployment();
        v1Deployment.setMetadata(meta);
        v1Deployment.setKind("Deployment");
        v1Deployment.setApiVersion(deploymentAddVo.getVersion());
        V1DeploymentSpec deploymentSpec = new V1DeploymentSpec();
        if (deploymentAddVo.getReplicas()==null){
            deploymentAddVo.setReplicas(1);
        }
        deploymentSpec.setReplicas(deploymentAddVo.getReplicas());
        V1LabelSelector selector = new V1LabelSelector();
        selector.setMatchLabels(labels);
        deploymentSpec.setSelector(selector);
        V1PodTemplateSpec templateSpec = new V1PodTemplateSpec();
        V1ObjectMeta v1ObjectMeta = new V1ObjectMeta();
        v1ObjectMeta.setLabels(labels);
        templateSpec.setMetadata(v1ObjectMeta);
        V1PodSpec v1PodSpec = new V1PodSpec();
        //set containers
        List<ContainerVo> containerVos = deploymentAddVo.getContainers();
        if (CollectionUtil.isEmpty(containerVos)){
            log.info("副本集镜像列表不能为空");
            return Boolean.FALSE;
        }
        List<V1Container> containers = new LinkedList<>();
        for (ContainerVo containerVo : containerVos) {
            //set volumes
            if (CollectionUtil.isNotEmpty(containerVo.getHostPath())){
                List<V1Volume> volumes = new LinkedList<>();
                List<ClaimHostVo> hostPaths = containerVo.getHostPath();
                for (ClaimHostVo path : hostPaths) {
                    V1Volume v1Volume = new V1Volume();
                    V1HostPathVolumeSource hostPathVolumeSource = new V1HostPathVolumeSource();
                    hostPathVolumeSource.setType(path.getType());
                    hostPathVolumeSource.setPath(path.getHostPath());
                    v1Volume.setHostPath(hostPathVolumeSource);
                    v1Volume.setName(path.getVolumeClaim());
                    volumes.add(v1Volume);
                }
                v1PodSpec.setVolumes(volumes);
            }
            V1Container v1Container = new V1Container();
            v1Container.setName(containerVo.getName());
            if (!containerVo.getImage().contains(":")){
                log.info("镜像地址需要携带版本号");
                return Boolean.FALSE;
            }
            v1Container.setImage(containerVo.getImage());
            v1Container.setImagePullPolicy(containerVo.getImagePullPolicy());
            if (CollectionUtil.isNotEmpty(containerVo.getHostPath())){
                List<V1VolumeMount> volumeMounts = new LinkedList<>();
                List<ClaimMountVo> mountPath = containerVo.getMountPath();
                for (ClaimMountVo path : mountPath) {
                    V1VolumeMount volumeMount = new V1VolumeMount();
                    volumeMount.setName(path.getVolumeClaim());
                    volumeMount.setMountPath(path.getMountPath());
                    volumeMounts.add(volumeMount);
                }
                v1Container.setVolumeMounts(volumeMounts);
            }
            V1EnvFromSource envFromSource = new V1EnvFromSource();
            if (StrUtil.isNotEmpty(deploymentAddVo.getConfigMap())){
                envFromSource.setConfigMapRef(new V1ConfigMapEnvSource().name(deploymentAddVo.getConfigMap()));
                v1Container.setEnvFrom(Collections.singletonList(envFromSource));
            }
            containers.add(v1Container);
        }
        v1PodSpec.setContainers(containers);
        //指定节点名称部署
        String nodeName = deploymentAddVo.getNodeName();
        if (StrUtil.isNotEmpty(nodeName)){
            v1PodSpec.setNodeName(nodeName);
        }
        templateSpec.setSpec(v1PodSpec);
        deploymentSpec.template(templateSpec);
        v1Deployment.setSpec(deploymentSpec);
        try {
            api.createNamespacedDeployment(deploymentAddVo.getNameSpace(), v1Deployment, "true", null, null, null);
            return Boolean.TRUE;
        }catch (ApiException e){
            log.error("副本集deployment部署失败");
            log.error("Status code: {}", e.getCode());
            log.error("Reason: {}", e.getResponseBody());
            return Boolean.FALSE;
        }
    }

    @Override
    public Boolean applyStatefulSet(StatefulSetAddVo statefulSetAddVo) {
        AppsV1Api api = new AppsV1Api();
        V1StatefulSet statefulSet = new V1StatefulSet();
        statefulSet.setKind("StatefulSet");
        V1ObjectMeta meta = new V1ObjectMeta();
        if (StrUtil.isEmpty(statefulSetAddVo.getName())){
            log.info("状态副本集名称不能为空");
            return Boolean.FALSE;
        }
        if (StrUtil.isEmpty(statefulSetAddVo.getNameSpace())){
            log.info("需要指定命名空间");
            return Boolean.FALSE;
        }
        meta.setName(statefulSetAddVo.getName());
        meta.setNamespace(statefulSetAddVo.getNameSpace());
        List<LabelsOrKvVo> labelVo = statefulSetAddVo.getLabels();
        if (CollectionUtil.isEmpty(labelVo)){
            log.info("至少需要输入一个标签");
            return Boolean.FALSE;
        }
        Map<String,String> labels =  new HashMap<>(labelVo.size());
        for (LabelsOrKvVo labelsOrKvVo : labelVo) {
            labels.put(labelsOrKvVo.getKey(), labelsOrKvVo.getValue());
        }
        meta.setLabels(labels);
        statefulSet.setMetadata(meta);
        V1StatefulSetSpec spec = new V1StatefulSetSpec();
        if (statefulSetAddVo.getReplicas()==null){
            statefulSetAddVo.setReplicas(1);
        }
        spec.setReplicas(statefulSetAddVo.getReplicas());
        V1LabelSelector selector = new V1LabelSelector();
        selector.setMatchLabels(labels);
        spec.setSelector(selector);
        spec.setServiceName(statefulSetAddVo.getName());

        V1PodTemplateSpec templateSpec = new V1PodTemplateSpec();
        V1ObjectMeta v1ObjectMeta = new V1ObjectMeta();
        v1ObjectMeta.setLabels(labels);
        templateSpec.setMetadata(v1ObjectMeta);
        V1PodSpec v1PodSpec = new V1PodSpec();
        //set containers
        List<ContainerVo> containerVos = statefulSetAddVo.getContainers();
        if (CollectionUtil.isEmpty(containerVos)){
            log.info("副本集镜像列表不能为空");
            return Boolean.FALSE;
        }
        List<V1Container> containers = new LinkedList<>();
        for (ContainerVo containerVo : containerVos) {
            //set volumes
            if (CollectionUtil.isNotEmpty(containerVo.getHostPath())){
                List<V1Volume> volumes = new LinkedList<>();
                List<ClaimHostVo> hostPaths = containerVo.getHostPath();
                for (ClaimHostVo path : hostPaths) {
                    V1Volume v1Volume = new V1Volume();
                    V1HostPathVolumeSource hostPathVolumeSource = new V1HostPathVolumeSource();
                    hostPathVolumeSource.setType(path.getType());
                    hostPathVolumeSource.setPath(path.getHostPath());
                    v1Volume.setHostPath(hostPathVolumeSource);
                    v1Volume.setName(path.getVolumeClaim());
                    volumes.add(v1Volume);
                }
                v1PodSpec.setVolumes(volumes);
            }
            V1Container v1Container = new V1Container();
            v1Container.setName(containerVo.getName());
            if (!containerVo.getImage().contains(":")){
                log.info("镜像地址需要携带版本号");
                return Boolean.FALSE;
            }
            v1Container.setImage(containerVo.getImage());
            v1Container.setImagePullPolicy(containerVo.getImagePullPolicy());
            if (CollectionUtil.isNotEmpty(containerVo.getHostPath())){
                List<V1VolumeMount> volumeMounts = new LinkedList<>();
                List<ClaimMountVo> mountPath = containerVo.getMountPath();
                for (ClaimMountVo path : mountPath) {
                    V1VolumeMount volumeMount = new V1VolumeMount();
                    volumeMount.setName(path.getVolumeClaim());
                    volumeMount.setMountPath(path.getMountPath());
                    volumeMounts.add(volumeMount);
                }
                v1Container.setVolumeMounts(volumeMounts);
            }
            V1EnvFromSource envFromSource = new V1EnvFromSource();
            if (StrUtil.isNotEmpty(statefulSetAddVo.getConfigMap())){
                envFromSource.setConfigMapRef(new V1ConfigMapEnvSource().name(statefulSetAddVo.getConfigMap()));
                v1Container.setEnvFrom(Collections.singletonList(envFromSource));
            }
            containers.add(v1Container);
        }
        v1PodSpec.setContainers(containers);
        //指定节点名称部署
        String nodeName = statefulSetAddVo.getNodeName();
        if (StrUtil.isNotEmpty(nodeName)){
            v1PodSpec.setNodeName(nodeName);
        }
        templateSpec.setSpec(v1PodSpec);
        spec.setTemplate(templateSpec);
        statefulSet.setSpec(spec);
        try {
            api.createNamespacedStatefulSet(statefulSetAddVo.getNameSpace(),statefulSet,"true",null,null,null);
        } catch (ApiException e) {
            log.error("状态副本集StatefulSet部署失败");
            log.error("Status code: {}", e.getCode());
            log.error("Reason: {}", e.getResponseBody());
            return Boolean.FALSE;
        }
        return Boolean.TRUE;
    }

    @Override
    public List<DeploymentVo> deployments(String nameSpace) {
        List<DeploymentVo> deploymentVos = new LinkedList<>();
        AppsV1Api api = new AppsV1Api();
        V1DeploymentList deployment;
        try {
            deployment = api.listNamespacedDeployment("xiaoshu","ture", false, null, null, null, 60, null,
                    null, 60, false);
        } catch (ApiException e) {
            e.printStackTrace();
            return deploymentVos;
        }
        if (deployment==null){
            return deploymentVos;
        }
        List<V1Deployment> items = deployment.getItems();
        if (CollectionUtil.isNotEmpty(items)){
            items.forEach(e->{
                V1DeploymentStatus status = e.getStatus();
                Integer readyReplicas = status.getReadyReplicas();
                Integer availableReplicas = status.getAvailableReplicas();
                String curStatus="期望副本数："+readyReplicas+"实际副本数："+availableReplicas;
                DeploymentVo deploymentVo = new DeploymentVo();
                deploymentVo.setStatus(curStatus);
                V1ObjectMeta metadata = e.getMetadata();
                String name = metadata.getName();
                deploymentVo.setName(name);
                OffsetDateTime dateTime = metadata.getCreationTimestamp();
                Date date = Date.from(dateTime.toLocalDateTime().toInstant(ZoneOffset.ofHours(8)));
                deploymentVo.setCreateTime(date);
                V1DeploymentSpec spec = e.getSpec();
                Integer replicas = spec.getReplicas();
                deploymentVo.setReplicas(replicas);
                V1PodTemplateSpec template = spec.getTemplate();
                List<V1Container> containers = template.getSpec().getContainers();
                List<ContainerVo> containerVos = new LinkedList<>();
                containers.forEach(c->{
                    ContainerVo containerVo = new ContainerVo();
                    String containerName = c.getName();
                    containerVo.setName(containerName);
                    String image = c.getImage();
                    containerVo.setImage(image);
                    String imagePullPolicy = c.getImagePullPolicy();
                    containerVo.setImagePullPolicy(imagePullPolicy);
                    containerVos.add(containerVo);
                });
                deploymentVo.setContainers(containerVos);
                deploymentVos.add(deploymentVo);
            });
        }
        return deploymentVos;
    }

    @Override
    public List<VolumesPvVo> volumes(String nameSpace) {
        List<VolumesPvVo> res = new LinkedList<>();
        CoreV1Api api = new CoreV1Api();
        V1PersistentVolumeList pvList = null;
        try {
            pvList = api.listPersistentVolume("true",false,null,null,null,null,null,null,60,false);
            List<V1PersistentVolume> items = pvList.getItems();
            if (CollectionUtil.isNotEmpty(items)){
                items.forEach(e->{
                    VolumesPvVo volumesPvVo = new VolumesPvVo();
                    volumesPvVo.setName(e.getMetadata().getName());
                    V1PersistentVolumeSpec spec = e.getSpec();
                    volumesPvVo.setHostPath(spec.getHostPath().getPath());
                    volumesPvVo.setHostPathType(spec.getHostPath().getType());
                    volumesPvVo.setCapacity(LocalFileUtil.fileShow(spec.getCapacity().get("storage").getNumber().longValue()));
                    volumesPvVo.setVolumeMode(spec.getVolumeMode());
                    volumesPvVo.setStorageClassName(spec.getStorageClassName());
                    volumesPvVo.setAccessModes(spec.getAccessModes().get(0));
                    OffsetDateTime dateTime = e.getMetadata().getCreationTimestamp();
                    Date date = Date.from(dateTime.toLocalDateTime().toInstant(ZoneOffset.ofHours(8)));
                    volumesPvVo.setCreateTime(date);
                    volumesPvVo.setPersistentVolumeReclaimPolicy(spec.getPersistentVolumeReclaimPolicy());
                    volumesPvVo.setStatus(e.getStatus().getPhase());
                    res.add(volumesPvVo);
                });
            }
        } catch (ApiException e) {
            System.err.println("Exception when calling CoreV1Api#listPersistentVolume");
            e.printStackTrace();
        }
        return res;
    }

    @Override
    public List<VolumesPvClaimVo> volumesClaim(String nameSpace) {
        List<VolumesPvClaimVo> res = new LinkedList<>();
        CoreV1Api api = new CoreV1Api();
        try {
            V1PersistentVolumeClaimList persistentVolumeClaim = api.listNamespacedPersistentVolumeClaim(nameSpace, "true", false, null, null, null,
                    null, null, null, 60, false);
            List<V1PersistentVolumeClaim> items = persistentVolumeClaim.getItems();
            if (CollectionUtil.isNotEmpty(items)){
                items.forEach(e->{
                    VolumesPvClaimVo pvClaimVo = new VolumesPvClaimVo();
                    pvClaimVo.setClaimName(e.getMetadata().getName());
                    pvClaimVo.setNameSpace(e.getMetadata().getNamespace());
                    OffsetDateTime dateTime = e.getMetadata().getCreationTimestamp();
                    Date date = Date.from(dateTime.toLocalDateTime().toInstant(ZoneOffset.ofHours(8)));
                    pvClaimVo.setCreateTime(date);
                    V1PersistentVolumeClaimSpec spec = e.getSpec();
                    pvClaimVo.setName(spec.getVolumeName());
                    pvClaimVo.setAccessModes(spec.getAccessModes().get(0));
                    pvClaimVo.setStorageClassName(spec.getStorageClassName());
                    pvClaimVo.setVolumeMode(spec.getVolumeMode());
                    V1PersistentVolumeClaimStatus status = e.getStatus();
                    pvClaimVo.setCapacity(LocalFileUtil.fileShow(status.getCapacity().get("storage").getNumber().longValue()));
                    pvClaimVo.setStatus(status.getPhase());
                    res.add(pvClaimVo);
                });
            }
        } catch (ApiException e) {
            e.printStackTrace();
        }
        return res;
    }

    @Override
    public Boolean addVolumesClaim(VolumesAddVo volumesAddVo) {
        CoreV1Api api = new CoreV1Api();
        V1PersistentVolumeClaim pvc = new V1PersistentVolumeClaim();
        // Set the metadata for the PVC
        V1ObjectMeta meta = new V1ObjectMeta();
        meta.setName(volumesAddVo.getClaimName());
        meta.setNamespace(volumesAddVo.getNameSpace());
        pvc.setMetadata(meta);
        pvc.setKind("PersistentVolumeClaim");
        // Set the spec for the PVC
        V1PersistentVolumeClaimSpec spec = new V1PersistentVolumeClaimSpec();
        //配置容量
        Quantity quantity = new Quantity(volumesAddVo.getCapacity());
        V1ResourceRequirements resources = new V1ResourceRequirements();
        resources.putRequestsItem("storage",quantity);
        spec.setResources(resources);
        //配置访问模式
        List<String> accessModes = new LinkedList<>();
        accessModes.add(volumesAddVo.getAccessModes());
        spec.setAccessModes(accessModes);
        //..
        spec.setVolumeMode(volumesAddVo.getVolumeMode());
        spec.setStorageClassName(volumesAddVo.getStorageClassName());
        spec.setVolumeName(volumesAddVo.getName());
        pvc.setSpec(spec);
        try {
            api.createNamespacedPersistentVolumeClaim(volumesAddVo.getNameSpace(), pvc, "true", null, null,null);
        } catch (ApiException e) {
            log.error("数据卷申请单PVC创建失败");
            log.error("Status code: {}", e.getCode());
            log.error("Reason: {}", e.getResponseBody());
            return Boolean.FALSE;
        }
        return Boolean.TRUE;
    }

    @Override
    public Boolean addVolumes(VolumesAddVo volumesAddVo) {
        CoreV1Api api = new CoreV1Api();
        V1PersistentVolume pv = new V1PersistentVolume();
        V1ObjectMeta meta = new V1ObjectMeta();
        meta.setName(volumesAddVo.getName());
        meta.setNamespace(volumesAddVo.getNameSpace());
        pv.setMetadata(meta);
        pv.setKind("PersistentVolume");
        //设置
        V1PersistentVolumeSpec spec = new V1PersistentVolumeSpec();
        List<String> accessModes = new LinkedList<>();
        accessModes.add(volumesAddVo.getAccessModes());
        spec.setAccessModes(accessModes);
        //设置挂载
        spec.setVolumeMode(volumesAddVo.getVolumeMode());
        spec.setStorageClassName(volumesAddVo.getStorageClassName());
        V1HostPathVolumeSource pathVolumeSource = new V1HostPathVolumeSource();
        pathVolumeSource.setPath(volumesAddVo.getHostPath());
        pathVolumeSource.setType(volumesAddVo.getHostPathType());
        spec.setHostPath(pathVolumeSource);
        //配置容量
        Map<String,Quantity> capacityMap = new HashMap<>();
        capacityMap.put("storage",new Quantity(volumesAddVo.getCapacity()));
        spec.setCapacity(capacityMap);
        pv.setSpec(spec);
        //回收策略
        spec.setPersistentVolumeReclaimPolicy(volumesAddVo.getPersistentVolumeReclaimPolicy());
        try {
            api.createPersistentVolume(pv,"ture",null,null,null);
        } catch (ApiException e) {
            log.error("数据卷PV创建失败");
            log.error("Status code: {}", e.getCode());
            log.error("Reason: {}", e.getResponseBody());
            return Boolean.FALSE;
        }
        return Boolean.TRUE;
    }

    @Override
    public List<ServiceAddVo> services(String nameSpace) {
        List<ServiceAddVo> res = new LinkedList<>();
        CoreV1Api api = new CoreV1Api();
        try {
            V1ServiceList listNamespacedService = api.listNamespacedService(nameSpace, "true", false, null, null, null, null, null, null, null, false);
            if (listNamespacedService==null){
                log.info("服务列表查询失败");
                return res;
            }
            List<V1Service> items = listNamespacedService.getItems();
            items.forEach(e->{
                ServiceAddVo serviceAddVo = new ServiceAddVo();
                V1ObjectMeta metadata = e.getMetadata();
                String name = metadata.getName();
                serviceAddVo.setName(name);
                Map<String, String> labels = metadata.getLabels();
                List<LabelsOrKvVo> curLabels = new LinkedList<>();
                if (CollectionUtil.isNotEmpty(labels)){
                    labels.forEach((k,v)->{
                        LabelsOrKvVo labelsOrKvVo = new LabelsOrKvVo();
                        labelsOrKvVo.setKey(k);
                        labelsOrKvVo.setValue(v);
                        curLabels.add(labelsOrKvVo);
                    });
                }
                serviceAddVo.setLabels(curLabels);
                OffsetDateTime dateTime = metadata.getCreationTimestamp();
                Date date = Date.from(dateTime.toLocalDateTime().toInstant(ZoneOffset.ofHours(8)));
                serviceAddVo.setCreateTime(date);
                String curNamespace = metadata.getNamespace();
                serviceAddVo.setNameSpace(curNamespace);
                V1ServiceSpec spec = e.getSpec();
                String type = spec.getType();
                serviceAddVo.setType(type);
                List<V1ServicePort> ports = spec.getPorts();
                List<ServicePortVo> curPorts = new LinkedList<>();
                if (CollectionUtil.isNotEmpty(ports)){
                    for (V1ServicePort port : ports) {
                        ServicePortVo servicePortVo = new ServicePortVo();
                        Integer nodePort = port.getNodePort();
                        servicePortVo.setNodePort(nodePort);
                        IntOrString targetPort = port.getTargetPort();
                        servicePortVo.setContainerPort(targetPort.getIntValue());
                        curPorts.add(servicePortVo);
                    }
                }
                serviceAddVo.setPorts(curPorts);
                res.add(serviceAddVo);
            });
        } catch (ApiException e) {
            e.printStackTrace();
        }
        return res;
    }

    @Override
    public Boolean applyService(ServiceAddVo serviceAddVo) {
        CoreV1Api api = new CoreV1Api();
        V1Service v1Service = new V1Service();
        List<LabelsOrKvVo> labelVo = serviceAddVo.getLabels();
        if (CollectionUtil.isEmpty(labelVo)){
            log.info("至少需要输入一个标签");
            return Boolean.FALSE;
        }
        Map<String,String> labels =  new HashMap<>(labelVo.size());
        for (LabelsOrKvVo labelsOrKvVo : labelVo) {
            labels.put(labelsOrKvVo.getKey(), labelsOrKvVo.getValue());
        }
        v1Service.setApiVersion(serviceAddVo.getVersion());
        v1Service.setKind("Service");
        V1ObjectMeta meta = new V1ObjectMeta();
        meta.setName(serviceAddVo.getName());
        meta.setNamespace(serviceAddVo.getNameSpace());
        meta.setLabels(labels);
        v1Service.setMetadata(meta);
        V1ServiceSpec serviceSpec = new V1ServiceSpec();
        serviceSpec.setSelector(labels);
        serviceSpec.setType(serviceAddVo.getType());

        List<ServicePortVo> voPorts = serviceAddVo.getPorts();
        if (CollectionUtil.isEmpty(voPorts)){
            return Boolean.FALSE;
        }
        List<V1ServicePort> ports = new LinkedList<>();
        for (ServicePortVo portVo : voPorts) {
            V1ServicePort v1ServicePort = new V1ServicePort();
            v1ServicePort.setPort(portVo.getContainerPort());
            v1ServicePort.setProtocol(portVo.getProtocol());
            IntOrString intOrString = new IntOrString(portVo.getContainerPort());
            v1ServicePort.setTargetPort(intOrString);
            v1ServicePort.setNodePort(portVo.getNodePort());
            ports.add(v1ServicePort);
        }
        serviceSpec.setPorts(ports);
        v1Service.setSpec(serviceSpec);
        try {
            api.createNamespacedService(serviceAddVo.getNameSpace(), v1Service, "true", null, null, null);
        }catch (ApiException e){
            log.error("Status code: {}", e.getCode());
            log.error("Reason: {}", e.getResponseBody());
            return Boolean.FALSE;
        }
        return Boolean.TRUE;
    }

    @Override
    public Boolean deleteService(DeleteK8sVo deleteK8sVo) {
        CoreV1Api api = new CoreV1Api();
        try {
            V1DeleteOptions deleteOptions = new V1DeleteOptions();
            deleteOptions.setPropagationPolicy(K8sConstants.DELETE_OPTION_FOREGROUND);
            api.deleteNamespacedService(deleteK8sVo.getName(),deleteK8sVo.getNameSpace(),null,null,null,false,null,deleteOptions);
        } catch (ApiException e) {
            log.error("Status code: {}", e.getCode());
            log.error("Reason: {}", e.getResponseBody());
            return Boolean.FALSE;
        }
        return Boolean.TRUE;
    }

    @Override
    public Boolean deleteDeployment(DeleteK8sVo deleteK8sVo) {
        AppsV1Api api = new AppsV1Api();
        try {
            V1DeleteOptions deleteOptions = new V1DeleteOptions();
            deleteOptions.setPropagationPolicy(K8sConstants.DELETE_OPTION_FOREGROUND);
            api.deleteNamespacedDeployment(deleteK8sVo.getName(), deleteK8sVo.getNameSpace(), "true", null, null, false, null, deleteOptions);
        } catch (ApiException e) {
            log.error("Status code: {}", e.getCode());
            log.error("Reason: {}", e.getResponseBody());
            return Boolean.FALSE;
        }
        return Boolean.TRUE;
    }

    @Override
    public Boolean addConfigMap(ConfigMapVo configMapVo) {
        CoreV1Api api = new CoreV1Api();
        V1ConfigMap configMap = new V1ConfigMap();
        Map<String, String> config = new HashMap<>();
        List<ConfigVo> configVos = configMapVo.getMaps();
        if (CollectionUtil.isNotEmpty(configVos)){
            for (ConfigVo configVo : configVos) {
                config.put(configVo.getKey(),configVo.getValue());
            }
        }
        configMap.setData(config);
        V1ObjectMeta metadata = new V1ObjectMeta();
        metadata.setName(configMapVo.getName());
        configMap.setMetadata(metadata);
        try {
            api.createNamespacedConfigMap(configMapVo.getNameSpace(), configMap, "true",null,null,null);
        } catch (ApiException e) {
            e.printStackTrace();
            log.error("Status code: {}", e.getCode());
            log.error("Reason: {}", e.getResponseBody());
            return Boolean.FALSE;
        }
        return Boolean.TRUE;
    }
}