【katacontainer基础知识】kata-container介绍与原理

WHY ?弥补了传统容器技术安全性的缺点，Kata Containers通过使用硬件虚拟化来达到容器隔离的目的。每一个container/pod 都是基于一个独立的kernel实例来作为一个轻量级的虚拟机。自从每一个container/pod运行与独立的虚拟机上，他们不再从宿主机内核上获取相应所有的权限。WHAT ?kata container...

张忠琳

19485人浏览 · 2019-01-15 11:32:35

张忠琳 · 2019-01-15 11:32:35 发布

WHY ?

弥补了传统容器技术安全性的缺点，Kata Containers通过使用硬件虚拟化来达到容器隔离的目的。每一个container/pod 都是基于一个独立的kernel实例来作为一个轻量级的虚拟机。自从每一个container/pod运行与独立的虚拟机上，他们不再从宿主机内核上获取相应所有的权限。

WHAT ?

kata containers是由OpenStack基金会管理的容器项目。kata containers整合了Intel的 Clear Containers 和 Hyper.sh 的 runV，能够支持不同平台的硬件，并符合Open Container Initiative规范，同时还可以兼容k8s的 CRI（Container Runtime Interface）接口规范。项目包含几个配套组件，即Runtime，Agent， Proxy，Shim等。项目已于6月份release了1.0版本。
从docker架构上看，kata-container和原来的runc是平级的。大家知道docker只是管理容器生命周期的框架，真正启动容器最早用的是LXC，然后是runc，现在也可以换成kata了。所以说kata-container可以当做docker的一个插件，启动kata-container可以通过docker命令。

kata container组件

1. Agent

Kata-agent运行在guest负责管理容器。Kata-agent的执行单元是定义了一系列命名空间的沙盒。每个VM可以运行多个容器，支持k8s一个pod运行多个容器的需求。不过目前docker中，kata-runtime只能一个pod一个容器。Kata-agent通过gRPC和其他kata组件通信。Kata-agent使用libcontainer管理容器的生命周期

2. Runtime

kata-runtime是一个OCI兼容的容器运行时，负责处理OCI运行时规范指定的所有命令并启动kata-shim实例

配置文件是/usr/share/defaults/kata-containers/configuration.toml

# XXX: WARNING: this file is auto-generated.
# XXX:
# XXX: Source file: "cli/config/configuration.toml.in"
# XXX: Project:
# XXX:   Name: Kata Containers
# XXX:   Type: kata

[hypervisor.qemu]
path = "/usr/bin/qemu-lite-system-x86_64"
kernel = "/usr/share/kata-containers/vmlinuz.container"
image = "/usr/share/kata-containers/kata-containers.img"
machine_type = "pc"

# Optional space-separated list of options to pass to the guest kernel.
# For example, use `kernel_params = "vsyscall=emulate"` if you are having
# trouble running pre-2.15 glibc.
#
# WARNING: - any parameter specified here will take priority over the default
# parameter value of the same name used to start the virtual machine.
# Do not set values here unless you understand the impact of doing so as you
# may stop the virtual machine from booting.
# To see the list of default parameters, enable hypervisor debug, create a
# container and look for 'default-kernel-parameters' log entries.
kernel_params = ""

# Path to the firmware.
# If you want that qemu uses the default firmware leave this option empty
firmware = ""

3. Proxy

默认使用virtio-serial和VM通信。VM可以运行多个容器进程。在使用virtio-serial的情况下，与每个进程相关联的I/O流需要在主机上多路复用和解复用。
Kata-proxy给多个kata-shim和kata-runtime客户端提供对kata-agent提供访问，它的主要作用是在每个kata-shim和kata-agent之间路由I/O流和信号。Kata-proxy连接到kata-agent的unix域套接字上，这个套接字是kata-proxy启动时kata-runtime提供的

4. Shim

runtime运行在宿主机上，不能直接监控运行在虚拟机里的进程，最多只能看到QEMU进程。kata-shim监控容器进程，处理容器的所有I/O流，包括stdout、stdin和stderr，以及转发所有的要发送出去的信号。
Kata-shim还有其他功能：

通过一个UNIX域套接字连接到kata-proxy。这个套接字在kata-runtime启动kata-shim的时候，由kata-runtime传给kata-shim，同时带上了containerID和execID，两个ID用来识别shim管理的是哪个容器。
读取VM内部容器进程的输出流和错误流
使用SignalProcessRequest API转发从reaper到kata-proxy的信号
监控终端修改，并使用grpc TtyWinResize API转发到kata-proxy

kata-runtime kata-env

[Meta]
Version = "1.0.18"

[Runtime]
Debug = false
Path = "/usr/bin/kata-runtime"
[Runtime.Version]
Semver = "1.3.1"
Commit = "258eae0"
OCI = "1.0.1"
[Runtime.Config]
Path = "/etc/kata-containers/configuration.toml"

[Hypervisor]
MachineType = "pc"
Version = "QEMU emulator version 2.11.0\nCopyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers"
Path = "/usr/bin/qemu-lite-system-x86_64"
BlockDeviceDriver = "virtio-scsi"
EntropySource = "/dev/urandom"
Msize9p = 8192
MemorySlots = 10
Debug = false
UseVSock = false

[Image]
Path = "/usr/share/kata-containers/kata-containers-image_clearlinux_1.3.1_agent_c7fdd324cda.img"

[Kernel]
Path = "/usr/share/kata-containers/vmlinuz-4.14.67.16-4.4.container"
Parameters = ""

[Initrd]
Path = ""

[Proxy]
Type = "kataProxy"
Version = "kata-proxy version 1.3.1-d364b2e"
Path = "/usr/libexec/kata-containers/kata-proxy"
Debug = false

[Shim]
Type = "kataShim"
Version = "kata-shim version 1.3.1-58f757d"
Path = "/usr/libexec/kata-containers/kata-shim"
Debug = false

[Agent]
Type = "kata"

[Host]
Kernel = "3.10.0-957.el7.x86_64"
Architecture = "amd64"
VMContainerCapable = true
SupportVSocks = false
[Host.Distro]
Name = "CentOS Linux"
Version = "7"
[Host.CPU]
Vendor = "GenuineIntel"
Model = "Intel(R) Core(TM) i7-6500U CPU @ 2.50GHz"

[Netmon]
Version = "kata-netmon version 1.3.1"
Path = "/usr/libexec/kata-containers/kata-netmon"
Debug = false
Enable = false

HOW ?

1. centos 7 安装docker

yum -y install yum-utils
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum -y install docker-ce

2. centos 7 安装kata-containers

source /etc/os-release
echo $VERSION_ID
VERSION_ID=$VERSION_ID yum-config-manager --add-repo \
"http://download.opensuse.org/repositories/home:/katacontainers:/release/CentOS_${VERSION_ID}/home:katacontainers:release.repo"
yum -y install kata-runtime kata-proxy kata-shim

3. 配置docker启动runtime参数

--add-runtime kata-runtime=/usr/bin/kata-runtime --default-runtime=kata-runtime

[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target docker-storage-setup.service
Wants=docker-storage-setup.service

[Service]
Type=notify
Environment=GOTRACEBACK=crash
ExecReload=/bin/kill -s HUP $MAINPID
Delegate=yes
KillMode=process
ExecStart=/usr/bin/dockerd -s overlay2 --storage-opt overlay2.override_kernel_check=1 \
          --add-runtime kata-runtime=/usr/bin/kata-runtime --default-runtime=kata-runtime \
          $DOCKER_OPTS \
          $DOCKER_STORAGE_OPTIONS \
          $DOCKER_NETWORK_OPTIONS \
          $DOCKER_DNS_OPTIONS \
          $INSECURE_REGISTRY
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
TimeoutStartSec=1min
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

4. 重启docker进程

systemctl daemon-reload && systemctl restart docker

启动容器，查看容器状态，docker stats，同时发现启动qemu进程

katacontainer kubernetes 集成安装

a. 使用 rpm 安装

http://download.opensuse.org/repositories/home:/katacontainers:/releases:/x86_64:/master/CentOS_7/x86_64/
http://download.opensuse.org/repositories/home:/katacontainers:/releases:/x86_64:/stable-1.3/CentOS_7/x86_64/4 安装

b. runc

不用 docker 跟 k8s 通信而是用 cri-o，同时支持普通容器和 kata 容器，要单独安装 runc
https://github.com/opencontainers/runc/releases/download/v1.0.0-rc8/runc.amd64

chmod +x runc.amd64
mv runc.amd64 /usr/bin/runc

c. 安装 crictl CLI

https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.13.0/crictl-v1.13.0-linux-amd64.tar.gz

d. crio

https://kubernetes.io/docs/setup/production-environment/container-runtimes/#cri-o

Prerequisites
modprobe overlay
modprobe br_netfilter

# Setup required sysctl params, these persist across reboots.
cat > /etc/sysctl.d/99-kubernetes-cri.conf <<EOF
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

sysctl --system

# Install prerequisites
yum-config-manager --add-repo=https://cbs.centos.org/repos/paas7-crio-311-candidate/x86_64/os/

# Install CRI-O
yum install --nogpgcheck cri-o

/etc/crio/crio.conf 可以使用命令 "crio config --default >/etc/crio/crio.conf" 创建，然后修改 registries 即可：

e. 配置 crio

修改 kata-runtime

# runtime_untrusted_workload is the OCI compatible runtime used for untrusted
# container workloads. This is an optional setting, except if
# default_container_trust is set to "untrusted".
runtime_untrusted_workload = "/usr/bin/kata-runtime"

修改镜像仓库

registries = [
"docker.io"
]

修改 cgroupfs 等

# cgroup_manager is the cgroup management implementation to be used
# for the runtime.
cgroup_manager = "cgroupfs"

修改 pause 地址

# pause_image is the image which we use to instantiate infra containers.
pause_image = "zhangzhonglin/pause-amd64:3.0"

修改 plugin_dir 路径

# plugin_dir is is where CNI plugin binaries are stored.
plugin_dir = "/opt/k8s/bin"

Start CRI-O systemctl start crio

f. kubelet 配置

添加参数 --container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///var/run/crio/crio.sock

测试启动一个 deployment 以注解 io.kubernetes.cri-o.TrustedSandbox: "false"：

spec:
replicas: 1
selector:
matchLabels:
app: wordpress-hostpath
tier: mysql-hostpath
strategy:
type: Recreate
template:
metadata:
annotations:
io.kubernetes.cri-o.TrustedSandbox: "false"

crictl image

# crictl images
IMAGE TAG IMAGE ID SIZE
docker.io/calico/cni v3.7.4 203668d151b2e 139MB
docker.io/calico/kube-controllers v3.7.4 e67ede28cc7e4 46.8MB
docker.io/calico/node v3.7.4 84b65b552a8f0 159MB
docker.io/coredns/coredns 1.5.0 7987f0908caf3 42.7MB
docker.io/zhangzhonglin/pause-amd64 3.0 99e59f495ffaa 753kB

发现启动 crio common 进程

启动 qemu 进程

总结：

这个镜像管理比较搓，下载到本地还是不可以，必须是 crio.cfg 配置中的仓库必须能通才可以，containerd 最起码可以管理镜像可以 load

安装 kata-runtime，先验证环境是否支持虚拟化环境，否则就白玩了

kata-runtime kata-env，中 VMContainerCapable = true 这个是支持虚拟化环境。

2. 使用 containerd 作为 CRI

下载地址：https://github.com/containerd/containerd/releases
解压至目录 /usr/bin 下

sudo systemctl start containerd

sudo systemctl status containerd

containerd config default 生成默认配置，位置/etc/containerd/config.toml

2.1 crictl配置

# cat /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: true

2.2 containerd配置

containerd config default > /etc/containerd/config.toml

# cat /etc/containerd/config.toml

root = "/var/lib/containerd"
state = "/run/containerd"
oom_score = 0

[grpc]
address = "/run/containerd/containerd.sock"
uid = 0
gid = 0

.........................................................................

2.2.1 kata-container cri 插件配置，对于 1.5+版本

plugins.cri.containerd.runtimes.runc: the runc, and it is the default runtime

plugins.cri.containerd.runtimes.kata: The function in containerd (reference the document here) where the dot-connected string io.containerd.kata.v2 is translated to containerd-shim-kata-v2

    [plugins.cri.containerd]
      no_pivot = false
    [plugins.cri.containerd.runtimes]
      [plugins.cri.containerd.runtimes.runc]
         runtime_type = "io.containerd.runc.v1"
         [plugins.cri.containerd.runtimes.runc.options]
           NoPivotRoot = false
           NoNewKeyring = false
           ShimCgroup = ""
           IoUid = 0
           IoGid = 0
           BinaryName = "runc"
           Root = ""
           CriuPath = ""
           SystemdCgroup = false
      [plugins.cri.containerd.runtimes.kata]
         runtime_type = "io.containerd.kata.v2"
      [plugins.cri.containerd.runtimes.katacli]
         runtime_type = "io.containerd.runc.v1"
         [plugins.cri.containerd.runtimes.katacli.options]
           NoPivotRoot = false
           NoNewKeyring = false
           ShimCgroup = ""
           IoUid = 0
           IoGid = 0
           BinaryName = "/usr/bin/kata-runtime"
           Root = ""
           CriuPath = ""
           SystemdCgroup = false

参考： https://github.com/kata-containers/documentation/blob/master/how-to/containerd-kata.md

2.3 kubelet配置添加如下：

--runtime-cgroups=/system.slice/containerd.service --container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock

镜像下载的问题

ctr cri load xxx.tar

参考：

https://github.com/kata-containers/runtime

katacontainer 原理架构

Overview

kata-container 和 runc是平级的。docker 管理容器生命周期的框架，而真正启动容器是 runc，可以换成 kata。kata-container可以当做 docker 的一个插件

参考：

https://github.com/kata-containers/documentation/blob/master/design/architecture.md

https://yq.aliyun.com/articles/692117

安装 https://github.com/kata-containers/documentation/blob/master/install/installing-with-kata-doc-to-script.md

向您推荐>>Eolink开发者社区

权威｜前沿｜技术｜干货｜国内首个API全生命周期开发者社区

更多推荐

深入理解 Mocha 测试框架：从零实现一个 Mocha

前言什么是自动化测试自动化测试在很多团队中都是Devops环节中很难执行起来的一个环节，主要原因在于测试代码的编写工作很难抽象，99%的场景都需要和业务强绑定，而且写测试代码的编写工作量往往比编写实际业务代码的工作量更多。在一些很多业务场景中投入产出比很低，适合写自动化测试的应该是那些中长期业务以及一些诸如组件一样的基础库。自动化测试是个比较大的概念，其中分类也比较多，比如单元测试，端对端测试，集

云原生

ELK实现containerd的容器日志采集展示【基于logging的全栈监测】

企业级ELK Stack构建介绍

云原生

(20200916 Solved)docker-compose up创建容器自动退出

问题描述如题，创建容器后自动退出了。并且docker start container无效解决方案原因是缺失了控制终端的配置，需要在docker-compose.yml中增加tty:true ，有时候这样也不行，需要再增加一个command:/bin/bash，命令不一定是这个，需要是一个不会退出的命令，然后用-d后台启动容器。Referencesdocker-compose启动容器后自动退出...