树莓派4BUbuntu server 20.04 Kubernetes-v1.17.x- Docker19.03 keadmv1.10.3部署错误一览

树莓派4BUbuntu server 20.04  keadmv1.10.3加入主节点报错

execute keadm command failed:  failed to exec 'bash -c sudo ln /etc/kubeedge/edgecore.service /etc/systemd/system/edgecore.service && sudo systemctl daemon-reload && sudo systemctl enable edgecore && sudo systemctl start edgecore', err: ln: failed to create hard link '/etc/systemd/system/edgecore.service': File exists
, err: exit status 1

解决办法删除'/etc/systemd/system/edgecore.service'重新加入节点，成功。

之后journalctl -u edgecore.service -xe 打印日志

报错

 Version: v1.10.3
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.205200  241646 server.go:103] Use node IP address from config: 192.168.137.74
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.205291  241646 sql.go:21] Begin to register twin db model
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.205673  241646 module.go:52] Module twin registered successfully
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.228239  241646 client.go:78] "Connecting to docker on the dockerEndpoint" endpoint="unix:///var/run/docker.sock"
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.228365  241646 client.go:97] "Start docker client with request timeout" timeout="0s"
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.262259  241646 docker_service.go:242] "Hairpin mode is set" hairpinMode=hairpin-veth
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.263310  241646 cni.go:204] "Error validating CNI config list" configList="{\n  \"name\": \"cbr0\",\n  \"cniVersion\": \"0.3.1\",\n  \"plugins\": [\n    {\n      \"type\": \"flannel\",\n     >
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.263411  241646 cni.go:239] "Unable to update cni config" err="no valid networks found in /etc/cni/net.d"
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.269385  241646 hostport_manager.go:72] "The binary conntrack is not installed, this can cause failures in network connection cleanup."
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.269521  241646 hostport_manager.go:72] "The binary conntrack is not installed, this can cause failures in network connection cleanup."
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.275158  241646 docker_service.go:257] "Docker cri networking managed by the network plugin" networkPluginName="kubernetes.io/no-op"
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.312914  241646 docker_service.go:263] "Docker Info" dockerInfo=&{ID:X4KE:OB4J:5JH2:IYG4:X2FR:G4LB:OOWP:FUNK:6L46:7LM5:7CD4:ONYQ Containers:0 ContainersRunning:0 ContainersPaused:0 Containers>
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.313017  241646 docker_service.go:278] "Setting cgroupDriver" cgroupDriver="cgroupfs"
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.313345  241646 edged.go:716] RemoteRuntimeEndpoint: "unix:///var/run/dockershim.sock", remoteImageEndpoint: "unix:///var/run/dockershim.sock"
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.313386  241646 edged.go:719] Starting the GRPC server for the docker CRI shim.
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.361424  241646 state_mem.go:36] "Initialized new in-memory state store"
Oct 10 16:09:23 node1 edgecore[241646]: W1010 16:09:23.361691  241646 edged.go:416] failed to get the edged's cgroup: mountpoint for memory not found. Edged system container metrics may be missing.
Oct 10 16:09:23 node1 edgecore[241646]: W1010 16:09:23.361945  241646 edged.go:423] failed to get the container runtime's cgroup: failed to get container name for docker process: mountpoint for memory not found. Runtime system container metrics >
Oct 10 16:09:23 node1 edgecore[241646]: W1010 16:09:23.560195  241646 sysinfo.go:203] Nodes topology is not available, providing CPU topology
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.670596  241646 kuberuntime_manager.go:245] "Container runtime initialized" containerRuntime="docker" version="19.03.12" apiVersion="1.40.0"
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.670736  241646 edged.go:600] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.693689  241646 container_manager_linux.go:280] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.693873  241646 container_manager_linux.go:285] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntim>
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.693953  241646 topology_manager.go:133] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.694000  241646 container_manager_linux.go:320] "Creating device plugin manager" devicePluginEnabled=false
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.694039  241646 state_mem.go:36] "Initialized new in-memory state store"
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.694294  241646 module.go:52] Module edged registered successfully
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.694335  241646 module.go:52] Module websocket registered successfully
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.694357  241646 module.go:52] Module eventbus registered successfully
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.694512  241646 metamanager.go:47] Begin to register metaManager db model
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.694691  241646 module.go:52] Module metaManager registered successfully
Oct 10 16:09:23 node1 edgecore[241646]: W1010 16:09:23.694721  241646 module.go:55] Module servicebus is disabled, do not register
Oct 10 16:09:23 node1 edgecore[241646]: W1010 16:09:23.694768  241646 module.go:55] Module edgestream is disabled, do not register
Oct 10 16:09:23 node1 edgecore[241646]: W1010 16:09:23.694791  241646 module.go:55] Module testManager is disabled, do not register
Oct 10 16:09:23 node1 edgecore[241646]: table `device` already exists, skip
Oct 10 16:09:23 node1 edgecore[241646]: table `device_attr` already exists, skip
Oct 10 16:09:23 node1 edgecore[241646]: table `device_twin` already exists, skip
Oct 10 16:09:23 node1 edgecore[241646]: table `sub_topics` already exists, skip
Oct 10 16:09:23 node1 edgecore[241646]: table `meta` already exists, skip
Oct 10 16:09:23 node1 edgecore[241646]: table `meta_v2` already exists, skip
Oct 10 16:09:23 node1 edgecore[241646]: table `target_urls` already exists, skip
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.698655  241646 core.go:46] starting module twin
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.698757  241646 core.go:46] starting module edged
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.698828  241646 process.go:114] Begin to sync sqlite
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.698881  241646 core.go:46] starting module websocket
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.698889  241646 edged.go:303] Starting edged...
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.698968  241646 core.go:46] starting module eventbus
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.699042  241646 core.go:46] starting module metaManager
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.699182  241646 http.go:40] tlsConfig InsecureSkipVerify true
Oct 10 16:09:23 node1 edgecore[241646]: E1010 16:09:23.699481  241646 csi_plugin.go:222] kubernetes.io/csi: CSIDriverLister not found on KubeletVolumeHost
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.699585  241646 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.699596  241646 common.go:97] start connect to mqtt server with client id: hub-client-sub-1665389363
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.699921  241646 common.go:99] client hub-client-sub-1665389363 isconnected: false
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.705799  241646 client.go:157] finish hub-client sub
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.705943  241646 common.go:97] start connect to mqtt server with client id: hub-client-pub-1665389363
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.705970  241646 common.go:99] client hub-client-pub-1665389363 isconnected: false
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.706533  241646 client.go:95] edge-hub-cli subscribe topic to $hw/events/upload/#
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.707352  241646 client.go:176] finish hub-client pub
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.707411  241646 eventbus.go:69] Init Sub And Pub Client for external mqtt broker tcp://127.0.0.1:1883 successfully
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.707714  241646 client.go:95] edge-hub-cli subscribe topic to $hw/events/device/+/state/update
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.708535  241646 client.go:95] edge-hub-cli subscribe topic to $hw/events/device/+/twin/+
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.709211  241646 client.go:95] edge-hub-cli subscribe topic to $hw/events/node/+/membership/get
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.709968  241646 client.go:95] edge-hub-cli subscribe topic to SYS/dis/upload_records
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.710532  241646 client.go:95] edge-hub-cli subscribe topic to +/user/#
Oct 10 16:09:23 node1 edgecore[241646]: I1010 16:09:23.711463  241646 client.go:103] list edge-hub-cli-topics status, no record, skip sync
Oct 10 16:09:24 node1 edgecore[241646]: I1010 16:09:24.162840  241646 cpu_manager.go:209] "Starting CPU manager" policy="none"
Oct 10 16:09:24 node1 edgecore[241646]: I1010 16:09:24.162909  241646 cpu_manager.go:210] "Reconciling" reconcilePeriod="0s"
Oct 10 16:09:24 node1 edgecore[241646]: I1010 16:09:24.162984  241646 state_mem.go:36] "Initialized new in-memory state store"
Oct 10 16:09:24 node1 edgecore[241646]: I1010 16:09:24.163639  241646 state_mem.go:88] "Updated default CPUSet" cpuSet=""
Oct 10 16:09:24 node1 edgecore[241646]: I1010 16:09:24.163709  241646 state_mem.go:96] "Updated CPUSet assignments" assignments=map[]
Oct 10 16:09:24 node1 edgecore[241646]: I1010 16:09:24.163743  241646 policy_none.go:49] "None policy: Start"
Oct 10 16:09:24 node1 edgecore[241646]: I1010 16:09:24.167587  241646 memory_manager.go:168] "Starting memorymanager" policy="None"
Oct 10 16:09:24 node1 edgecore[241646]: I1010 16:09:24.167715  241646 state_mem.go:35] "Initializing new in-memory state store"
Oct 10 16:09:24 node1 edgecore[241646]: I1010 16:09:24.168271  241646 state_mem.go:75] "Updated machine memory state"
Oct 10 16:09:24 node1 edgecore[241646]: E1010 16:09:24.169212  241646 edged.go:824] Failed to start container manager, err: system validation failed - Following Cgroup subsystem not mounted: [memory]
Oct 10 16:09:24 node1 edgecore[241646]: E1010 16:09:24.169265  241646 edged.go:307] initialize module error: system validation failed - Following Cgroup subsystem not mounted: [memory]
Oct 10 16:09:24 node1 systemd[1]: edgecore.service: Main process exited, code=exited, status=1/FAILURE

查看错误日志

Oct 10 21:40:05 node1 edgecore[466671]: I1010 21:40:05.212882  466671 server.go:76] Version: v1.10.3
Oct 10 21:40:05 node1 edgecore[466671]: I1010 21:40:05.225448  466671 server.go:103] Use node IP address from config: 192.168.137.74
Oct 10 21:40:05 node1 edgecore[466671]: I1010 21:40:05.225600  466671 sql.go:21] Begin to register twin db model
Oct 10 21:40:05 node1 edgecore[466671]: I1010 21:40:05.226142  466671 module.go:52] Module twin registered successfully
Oct 10 21:40:05 node1 edgecore[466671]: I1010 21:40:05.249774  466671 client.go:78] "Connecting to docker on the dockerEndpoint" endpoint="unix:///var/run/docker.sock"
Oct 10 21:40:05 node1 edgecore[466671]: I1010 21:40:05.249920  466671 client.go:97] "Start docker client with request timeout" timeout="0s"
Oct 10 21:40:05 node1 edgecore[466671]: I1010 21:40:05.284696  466671 docker_service.go:242] "Hairpin mode is set" hairpinMode=hairpin-veth
Oct 10 21:40:05 node1 edgecore[466671]: I1010 21:40:05.285558  466671 cni.go:204] "Error validating CNI config list" configList="{\n  \"name\": \"cbr0\",\n  \"cniVersion\": \"0.3.1\",\n  \"plugins\": [\n    {\n      \"type\": \"flannel\",\n      \"delegate\": {\n        \"hairpinMode\": true,\n        \"isDefaultGateway\": true\n      }\n    >
Oct 10 21:40:05 node1 edgecore[466671]: I1010 21:40:05.285634  466671 cni.go:239] "Unable to update cni config" err="no valid networks found in /etc/cni/net.d"
Oct 10 21:40:05 node1 edgecore[466671]: I1010 21:40:05.291651  466671 hostport_manager.go:72] "The binary conntrack is not installed, this can cause failures in network connection cleanup."
Oct 10 21:40:05 node1 edgecore[466671]: I1010 21:40:05.291816  466671 hostport_manager.go:72] "The binary conntrack is not installed, this can cause failures in network connection cleanup."
Oct 10 21:40:05 node1 edgecore[466671]: I1010 21:40:05.298601  466671 docker_service.go:257] "Docker cri networking managed by the network plugin" networkPluginName="kubernetes.io/no-op"
Oct 10 21:40:05 node1 edgecore[466671]: I1010 21:40:05.337487  466671 docker_service.go:263] "Docker Info" dockerInfo=&{ID:X4KE:OB4J:5JH2:IYG4:X2FR:G4LB:OOWP:FUNK:6L46:7LM5:7CD4:ONYQ Containers:0 ContainersRunning:0 ContainersPaused:0 ContainersStopped:0 Images:0 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] >
Oct 10 21:40:05 node1 edgecore[466671]: E1010 21:40:05.337737  466671 edged.go:279] init new edged error, misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"

vim /etc/docker/daemon.jsn

# 把native.cgroupdriver=systemd 修改成 native.cgroupdriver=cgroupfs

docker info|grep "Cgroup Driver" 

#检查 是否输出 Cgroup Driver: cgroupfs

#重启docker 并且查看docker的状态

systemctl restart docker

systemctl status docker

docker 运行正常

vim /var/lib/kubelet/kubeadm-flags.env

把其中的systemd 也改成cgroupfs再启动下kubelet，重新载入配置

systemctl daemon-reload
systemctl restart kubelet
systemctl enable kubelet
systemctl start kubelet
systemctl status kubelet

查看日志发现 

level=warning msg="Your kernel does not support cgroup rt runtime"
master dockerd[22355]: time="2022-10-10T21:51:48.629194761+08:00" level=warning msg="Your kernel does not support cgroup blkio weight"
master dockerd[22355]: time="2022-10-10T21:51:48.629205446+08:00" level=warning msg="Your kernel does not support cgroup blkio weight_device"

主节点docker，此条警告可以忽略，不影响。

再次加入kubeedge，发现报错

Failed to start container manager, err: system validation failed - Following Cgroup subsystem not mounted: [memory]
2388 edged.go:307] initialize module error: system validation failed - Following Cgroup subsystem not mounted: [memory]
ess exited, code=exited, status=1/FAILURE

日志显示的是内存资源的cgroups机制没有开启。
查看可用的cgroup:

root@node1:~/keadm-v1.10.3-linux-arm64/keadm# cat /proc/cgroups
#subsys_name    hierarchy       num_cgroups     enabled
cpuset  8       1       1
cpu     2       102     1
cpuacct 2       102     1
blkio   7       102     1
memory  0       112     0
devices 6       102     1
freezer 10      2       1
net_cls 3       1       1
perf_event      11      1       1
net_prio        3       1       1
hugetlb 9       1       1
pids    4       111     1
rdma    5       1       1

修改启动参数

sudo vim /boot/firmware/cmdline.txt

添加以下内容：

cgroup_memory=1 cgroup_enable=memory

添加在同一行的最后面,接着内容后空格后添加, 不要换行

#重启机器配置生效

reboot 

#查看edgecore启动情况
ps aux|grep edgecore
root@node1:~/keadm-v1.10.3-linux-arm64/keadm# ps aux|grep edgecore
root         697  5.4  1.9 1657340 77492 ?       Ssl  23:00   0:52 /usr/local/bin/edgecore
root        2563  0.0  0.0   7696   744 pts/0    S+   23:16   0:00 grep --color=auto edgecore

#查看启动状态
sudo systemctl status edgecore

 edgecore.service
     Loaded: loaded (/etc/systemd/system/edgecore.service; bad; vendor preset: enabled)
     Active: active (running) since Mon 2022-10-10 23:00:25 CST; 16min ago
   Main PID: 697 (edgecore)
      Tasks: 13 (limit: 4435)
     Memory: 88.4M
     CGroup: /system.slice/edgecore.service
             └─697 /usr/local/bin/edgecore

#查看docker开机启动状态 enabled:开启, disabled:关闭
systemctl is-enabled edgecore

#设置开机自启动
systemctl enable edgecore

Ubuntu18.04+树莓派Ubuntu20.04部署kubeedge v1.10.3主要参考帖子

(39条消息) Ubuntu20.04+树莓派部署Kubeedge v1.6.1_YsyNoobCoder的博客-CSDN博客_kubeedge 树莓派(39条消息) KubeEdge 超详细部署记录&问题记载6.28_Jason-Ning的博客-CSDN博客

验证环节

(39条消息) K8S v1.17.17&KubeEdge v1.7部署指南+kubeedge-counter-demo示例_cike90的博客-CSDN博客

常用指令
#查看 systemd托管列表
systemctl list-unit-files
systemctl start cloudcore.service && systemctl enable cloudcore.service

#查看节点状态
kubectl get node
kubectl get po -A
kubectl get po -n kube-system
kubectl get po -o wide

#查看网络状态
yum install net-tools -y
netstat -ntlp

#查看docker状态docker
docker images | grep kubeedge-pi-counter

#传文件
scp kubeedge-pi-counter.tar 192.168.xx.xx:/root/
tmp目录下的文件

注意事项：
1、一般情况下，使用完成尽量挂起虚拟机，不然容器停掉，启动后node会处于not reday状态。
2、如果node处于not reday状态了，执行一下重启脚本，执行命令如下：

bash restart_kubeedge.sh

脚本内容：
#!/bin/bash
for i in `docker ps -a | awk '{print $1}' | grep -v CONT`;do docker restart $i;done
for i in `docker ps -a | awk '{print $1}' | grep -v CONT`;do docker restart $i;done

 (39条消息) 【云原生1】零基础 k8s + kubeedge v1.5.0 部署_烈日松饼的博客-CSDN博客_kubeedge和k8s

（主要） 

 KubeEdge 部署记录 | 生命不息 折腾不止 (konata.tech)

排除错误：

kubernetes.io/csi: CSIDriverLister not found on KubeletVolumeHost

 (39条消息) KubeEdge1.10从零开始详细搭建教程_孟诗远msy的博客-CSDN博客_kubeedge搭建