k3s完全兼容k8s，我解释一下是为什么：

因为k3s的底层数据结构（写入etcd/mysql/sqlite/dsqlite数据库的格式），是完全照搬k8s的数据结构。api完全兼容。 对外表现相同，底层数据相同，只是实现方式不同（举个例子： 一个爆炒大虾，一个油焖虾）。

我认为 k3s/k8s数据结构的3个用途：

• 缓存从api中解析出来的参数，为了后面的处理逻辑更方便的去引用api的参数。
• 方便响应api时，用marshal() 去生成json/protobuf序列化的数据
• 方便写入数据库(etcd/mysql/sqlite)时，序列化为有序数据块

在k3s/k8s中，deploy、pod、service、Ingress、namespace、configmap等被成为资源。
yaml文件在定义一组(多组)资源的规格/约束条件的集合。

1 资源类型：

type TypeMeta struct {
        // rest api操作的资源类型：deployment/pod/servce/ingress/ns/pv/pvc
        Kind string `json:"kind,omitempty" protobuf:"bytes,1,opt,name=kind"`
        
        // APIVersion defines the versioned schema of this representation of an object.
        // Servers should convert recognized schemas to the latest internal value, and
        // may reject unrecognized values.
        APIVersion string `json:"apiVersion,omitempty" protobuf:"bytes,2,opt,name=apiVersion"`
}

2 资源元数据：

资源元数据：与api相关、与调度器相关、与数据库相关。 什么叫相关？ 直白点说：就是被读写

type ObjectMeta struct {
        // 在一个命名空间范围中，资源的名字(pod-https、pod-http、pod-web、sevice-reds、confiigmap-secrets-web)。  例如某个村里面(命名空间)不允许人重名，张三、李四
        Name string `json:"name,omitempty" protobuf:"bytes,1,opt,name=name"`

		// 在创建yaml文件时，可以指定资源名称、也可以不指定资源名称（不指定时则 用资源类型-随机字符拼接，例如pod-nx7jsda，service-hlayjk0a）,此变量就是k3s/k8s生成的默认资源名称
        GenerateName string `json:"generateName,omitempty" protobuf:"bytes,2,opt,name=generateName"`

		// 命名空间名称，例如default、filecoin、ingress-nginx。在同一个命名空间内，所有资源可互相感知存在。 在不同命名空间中，资源之间不能直接感知到其它命名空间的资源。
		Namespace string `json:"namespace,omitempty" protobuf:"bytes,3,opt,name=namespace"`

		// url地址，client可以通过访问此地址，来只读到资源的内容。  对应另外一个命令是读写，kubectl edit pod pod-name
		// k3s/k8s在1.21 release会删除此变量（因为有读写api、(历史原因存在的)只读api显得很鸡肋），无视它即可。
		SelfLink string `json:"selfLink,omitempty" protobuf:"bytes,4,opt,name=selfLink"`

		// k8s server在最终成功创建了资源后，会生成一个uuid给这资源，生成后，在资源的整个生命周期中不允许被修改。 （你可以理解为是身份证编号）
		UID types.UID `json:"uid,omitempty" protobuf:"bytes,5,opt,name=uid,casttype=k8s.io/kubernetes/pkg/types.UID"`

		// k3s/k8s 兼容多个api版本，即http/https rest api接口存在多个版本的路由，和多个版本对应的参数逻辑解析程序段
		// apiVersion: v1
		// apiVersion: v2
		// apps/v1beta1
		ResourceVersion string `json:"resourceVersion,omitempty" protobuf:"bytes,6,opt,name=resourceVersion"`

		// 流水号
		Generation int64 `json:"generation,omitempty" protobuf:"varint,7,opt,name=generation"`

		// 资源被创建时的时间戳
		CreationTimestamp Time `json:"creationTimestamp,omitempty" protobuf:"bytes,8,opt,name=creationTimestamp"`

		// 资源被删除时的时间戳，例如kubectl delete pods pod-name 命令被执行到时的时间戳
		DeletionTimestamp *Time `json:"deletionTimestamp,omitempty" protobuf:"bytes,9,opt,name=deletionTimestamp"`

		// 资源被删除时的宽限(给资源发让其退出的信号，等待资源关闭相关的系统资源后退出)时间（单位：秒），超过宽限时间则k3s/k8s会主动杀死资源
		DeletionGracePeriodSeconds *int64 `json:"deletionGracePeriodSeconds,omitempty" protobuf:"varint,10,opt,name=deletionGracePeriodSeconds"`
	
		// 给资源打标签： kubectl label node cka-node  disktype=gpu
		// 查看资源标签： kubectl get node --show-labels
		// 标签用途：用于更精细的去调度资源到指定的node，是k3/k8s为运维通讯提供的(管理/调度)扩展功能
		Labels map[string]string `json:"labels,omitempty" protobuf:"bytes,11,rep,name=labels"`

		// 注解：
		//Annotations 允许在 Kubernetes 资源上附加任意的非标识性元数据，用来记录资源的一些属性。这些元数据主要是给工具或者库来提取信息，一般不会直接开放给用户。绝大多数基于 Kubernetes 的开源项目都不依赖于 DB，完全可以利用 Kubernetes 的能力，满足对 DB 的需求。对于需要持久化的数据，除了定义 CRD，另一种通用的做法就是将数据存储在 annotation 中。
		//由于 annotation 的定位是 Kubernetes 资源上附加任意的非标识性元数据，除了在 key 上有跟 label key 完全一样的限制外，在 value 上没有任何限制：可长可短，可结构化可非结构化，可包含任意字符。
		Annotations map[string]string `json:"annotations,omitempty" protobuf:"bytes,12,rep,name=annotations"`

		// 包含足够的信息来让你确定一个对象，举个例子：你妈妈喊你吃饭时叫你小名：小张、狗子，你就知道是在叫你，而不用叫你的全名：张三丰、赵二狗
		OwnerReferences []OwnerReference `json:"ownerReferences,omitempty" patchStrategy:"merge" patchMergeKey:"uid" protobuf:"bytes,13,rep,name=ownerReferences"`

		// 资源在被删除之后，会写一个记录在这里切片中
		Finalizers []string `json:"finalizers,omitempty" patchStrategy:"merge" protobuf:"bytes,14,rep,name=finalizers"`

		// 资源属于的集群的名称。例如：易春雷老师(资源)是第二初级中学（集群名称）的优秀班主任， 
		ClusterName string `json:"clusterName,omitempty" protobuf:"bytes,15,opt,name=clusterName"`

		// 
		ManagedFields []ManagedFieldsEntry `json:"managedFields,omitempty" protobuf:"bytes,17,rep,name=managedFields"`
}


type ManagedFieldsEntry struct {
        // Manager is an identifier of the workflow managing these fields.
        Manager string `json:"manager,omitempty" protobuf:"bytes,1,opt,name=manager"`
        // Operation is the type of operation which lead to this ManagedFieldsEntry being created.
        // The only valid values for this field are 'Apply' and 'Update'.
        Operation ManagedFieldsOperationType `json:"operation,omitempty" protobuf:"bytes,2,opt,name=operation,casttype=ManagedFieldsOperationType"`
        // APIVersion defines the version of this resource that this field set
        // applies to. The format is "group/version" just like the top-level
        // APIVersion field. It is necessary to track the version of a field
        // set because it cannot be automatically converted.
        APIVersion string `json:"apiVersion,omitempty" protobuf:"bytes,3,opt,name=apiVersion"`
        // Time is timestamp of when these fields were set. It should always be empty if Operation is 'Apply'
        // +optional
        Time *Time `json:"time,omitempty" protobuf:"bytes,4,opt,name=time"`

        // Fields is tombstoned to show why 5 is a reserved protobuf tag.
        //Fields *Fields `json:"fields,omitempty" protobuf:"bytes,5,opt,name=fields,casttype=Fields"`

        // FieldsType is the discriminator for the different fields format and version.
        // There is currently only one possible value: "FieldsV1"
        FieldsType string `json:"fieldsType,omitempty" protobuf:"bytes,6,opt,name=fieldsType"`
        // FieldsV1 holds the first JSON version format as described in the "FieldsV1" type.
        // +optional
        FieldsV1 *FieldsV1 `json:"fieldsV1,omitempty" protobuf:"bytes,7,opt,name=fieldsV1"`

        // Subresource is the name of the subresource used to update that object, or
        // empty string if the object was updated through the main resource. The
        // value of this field is used to distinguish between managers, even if they
        // share the same name. For example, a status update will be distinct from a
        // regular update using the same manager name.
        // Note that the APIVersion field is not related to the Subresource field and
        // it always corresponds to the version of the main resource.
        Subresource string `json:"subresource,omitempty" protobuf:"bytes,8,opt,name=subresource"`
}

先写到这里，晚点补充其它数据结构。

3 服务：

type Service struct {          
        metav1.TypeMeta `json:",inline"`
      
        metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`
  
        // 存放servince.yaml文件定义的内容，被解析后的值
        Spec ServiceSpec `json:"spec,omitempty" protobuf:"bytes,2,opt,name=spec"`
      
        // 服务/ingress的实时状态
        Status ServiceStatus `json:"status,omitempty" protobuf:"bytes,3,opt,name=status"`
} 

type ServicePort struct {
        // 服务对应的端口的(唯一)名字
        Name string `json:"name,omitempty" protobuf:"bytes,1,opt,name=name"`

        // The IP protocol for this port. Supports "TCP", "UDP", and "SCTP".
        // Default is TCP.
        // +default="TCP"
        Protocol Protocol `json:"protocol,omitempty" protobuf:"bytes,2,opt,name=protocol,casttype=Protocol"`

        // The application protocol for this port.
        // This field follows standard Kubernetes label syntax.
        // Un-prefixed names are reserved for IANA standard service names (as per
        AppProtocol *string `json:"appProtocol,omitempty" protobuf:"bytes,6,opt,name=appProtocol"`

        // The port that will be exposed by this service.
        Port int32 `json:"port" protobuf:"varint,3,opt,name=port"`

        // 服务访问pod的端口
        TargetPort intstr.IntOrString `json:"targetPort,omitempty" protobuf:"bytes,4,opt,name=targetPort"`

        // linux node 真实port
        NodePort int32 `json:"nodePort,omitempty" protobuf:"varint,5,opt,name=nodePort"`
}


type ServiceSpec struct {
	// 本服务暴露的端口列表
	Ports []ServicePort `json:"ports,omitempty" patchStrategy:"merge" patchMergeKey:"port" protobuf:"bytes,1,rep,name=ports"`

	// 服务选择器：一组标签，用于匹配符合标签的资源
	Selector map[string]string `json:"selector,omitempty" protobuf:"bytes,2,rep,name=selector"`

	// 随机ip地址（kubectl cli未指定的话），指定本服务的ip地址
	ClusterIP string `json:"clusterIP,omitempty" protobuf:"bytes,3,opt,name=clusterIP"`

	// 随机ip地址列表（kubectl cli未指定的话），指定本服务的ip（多个）地址
	ClusterIPs []string `json:"clusterIPs,omitempty" protobuf:"bytes,18,opt,name=clusterIPs"`

	// 服务被暴露的方式：ClusterIP/NodePort/LoadBalancer
	Type ServiceType `json:"type,omitempty" protobuf:"bytes,4,opt,name=type,casttype=ServiceType"`

	// externalIPs is a list of IP addresses for which nodes in the cluster
	// will also accept traffic for this service.  These IPs are not managed by
	// Kubernetes.  The user is responsible for ensuring that traffic arrives
	// at a node with this IP.  A common example is external load-balancers
	// that are not part of the Kubernetes system.
	// +optional
	ExternalIPs []string `json:"externalIPs,omitempty" protobuf:"bytes,5,rep,name=externalIPs"`

	// Supports "ClientIP" and "None". Used to maintain session affinity.
	// Enable client IP based session affinity.
	// Must be ClientIP or None.
	// Defaults to None.
	// More info: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies
	// +optional
	SessionAffinity ServiceAffinity `json:"sessionAffinity,omitempty" protobuf:"bytes,7,opt,name=sessionAffinity,casttype=ServiceAffinity"`

	// Only applies to Service Type: LoadBalancer
	// LoadBalancer will get created with the IP specified in this field.
	// This feature depends on whether the underlying cloud-provider supports specifying
	// the loadBalancerIP when a load balancer is created.
	// This field will be ignored if the cloud-provider does not support the feature.
	// +optional
	LoadBalancerIP string `json:"loadBalancerIP,omitempty" protobuf:"bytes,8,opt,name=loadBalancerIP"`

	// If specified and supported by the platform, this will restrict traffic through the cloud-provider
	// load-balancer will be restricted to the specified client IPs. This field will be ignored if the
	// cloud-provider does not support the feature."
	// More info: https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/
	// +optional
	LoadBalancerSourceRanges []string `json:"loadBalancerSourceRanges,omitempty" protobuf:"bytes,9,opt,name=loadBalancerSourceRanges"`

	// externalName is the external reference that discovery mechanisms will
	// return as an alias for this service (e.g. a DNS CNAME record). No
	// proxying will be involved.  Must be a lowercase RFC-1123 hostname
	// (https://tools.ietf.org/html/rfc1123) and requires `type` to be "ExternalName".
	// +optional
	ExternalName string `json:"externalName,omitempty" protobuf:"bytes,10,opt,name=externalName"`

	// externalTrafficPolicy denotes if this Service desires to route external
	// traffic to node-local or cluster-wide endpoints. "Local" preserves the
	// client source IP and avoids a second hop for LoadBalancer and Nodeport
	// type services, but risks potentially imbalanced traffic spreading.
	// "Cluster" obscures the client source IP and may cause a second hop to
	// another node, but should have good overall load-spreading.
	// +optional
	ExternalTrafficPolicy ServiceExternalTrafficPolicyType `json:"externalTrafficPolicy,omitempty" protobuf:"bytes,11,opt,name=externalTrafficPolicy"`

	// 服务监控侦测(节点)端口，只在LoadBalancer / externalTrafficPolicy 模式时有效
	HealthCheckNodePort int32 `json:"healthCheckNodePort,omitempty" protobuf:"bytes,12,opt,name=healthCheckNodePort"`

	// publishNotReadyAddresses indicates that any agent which deals with endpoints for this
	// Service should disregard any indications of ready/not-ready.
	// The primary use case for setting this field is for a StatefulSet's Headless Service to
	// propagate SRV DNS records for its Pods for the purpose of peer discovery.
	// The Kubernetes controllers that generate Endpoints and EndpointSlice resources for
	// Services interpret this to mean that all endpoints are considered "ready" even if the
	// Pods themselves are not. Agents which consume only Kubernetes generated endpoints
	// through the Endpoints or EndpointSlice resources can safely assume this behavior.
	// +optional
	PublishNotReadyAddresses bool `json:"publishNotReadyAddresses,omitempty" protobuf:"varint,13,opt,name=publishNotReadyAddresses"`

	// sessionAffinityConfig contains the configurations of session affinity.
	// +optional
	SessionAffinityConfig *SessionAffinityConfig `json:"sessionAffinityConfig,omitempty" protobuf:"bytes,14,opt,name=sessionAffinityConfig"`

	// TopologyKeys is tombstoned to show why 16 is reserved protobuf tag.
	//TopologyKeys []string `json:"topologyKeys,omitempty" protobuf:"bytes,16,opt,name=topologyKeys"`

	// IPFamily is tombstoned to show why 15 is a reserved protobuf tag.
	// IPFamily *IPFamily `json:"ipFamily,omitempty" protobuf:"bytes,15,opt,name=ipFamily,Configcasttype=IPFamily"`

	// IPFamilies is a list of IP families (e.g. IPv4, IPv6) assigned to this
	// service. This field is usually assigned automatically based on cluster
	// configuration and the ipFamilyPolicy field. If this field is specified
	// manually, the requested family is available in the cluster,
	// and ipFamilyPolicy allows it, it will be used; otherwise creation of
	// the service will fail. This field is conditionally mutable: it allows
	// for adding or removing a secondary IP family, but it does not allow
	// changing the primary IP family of the Service. Valid values are "IPv4"
	// and "IPv6".  This field only applies to Services of types ClusterIP,
	// NodePort, and LoadBalancer, and does apply to "headless" services.
	// This field will be wiped when updating a Service to type ExternalName.
	//
	// This field may hold a maximum of two entries (dual-stack families, in
	// either order).  These families must correspond to the values of the
	// clusterIPs field, if specified. Both clusterIPs and ipFamilies are
	// governed by the ipFamilyPolicy field.
	// +listType=atomic
	// +optional
	IPFamilies []IPFamily `json:"ipFamilies,omitempty" protobuf:"bytes,19,opt,name=ipFamilies,casttype=IPFamily"`

	// IPFamilyPolicy represents the dual-stack-ness requested or required by
	// this Service. If there is no value provided, then this field will be set
	// to SingleStack. Services can be "SingleStack" (a single IP family),
	// "PreferDualStack" (two IP families on dual-stack configured clusters or
	// a single IP family on single-stack clusters), or "RequireDualStack"
	// (two IP families on dual-stack configured clusters, otherwise fail). The
	// ipFamilies and clusterIPs fields depend on the value of this field. This
	// field will be wiped when updating a service to type ExternalName.
	// +optional
	IPFamilyPolicy *IPFamilyPolicyType `json:"ipFamilyPolicy,omitempty" protobuf:"bytes,17,opt,name=ipFamilyPolicy,casttype=IPFamilyPolicyType"`

	// allocateLoadBalancerNodePorts defines if NodePorts will be automatically
	// allocated for services with type LoadBalancer.  Default is "true". It
	// may be set to "false" if the cluster load-balancer does not rely on
	// NodePorts.  If the caller requests specific NodePorts (by specifying a
	// value), those requests will be respected, regardless of this field.
	// This field may only be set for services with type LoadBalancer and will
	// be cleared if the type is changed to any other type.
	// This field is beta-level and is only honored by servers that enable the ServiceLBNodePortControl feature.
	// +featureGate=ServiceLBNodePortControl
	// +optional
	AllocateLoadBalancerNodePorts *bool `json:"allocateLoadBalancerNodePorts,omitempty" protobuf:"bytes,20,opt,name=allocateLoadBalancerNodePorts"`

	// loadBalancerClass is the class of the load balancer implementation this Service belongs to.
	// If specified, the value of this field must be a label-style identifier, with an optional prefix,
	// e.g. "internal-vip" or "example.com/internal-vip". Unprefixed names are reserved for end-users.
	// This field can only be set when the Service type is 'LoadBalancer'. If not set, the default load
	// balancer implementation is used, today this is typically done through the cloud provider integration,
	// but should apply for any default implementation. If set, it is assumed that a load balancer
	// implementation is watching for Services with a matching class. Any default load balancer
	// implementation (e.g. cloud providers) should ignore Services that set this field.
	// This field can only be set when creating or updating a Service to type 'LoadBalancer'.
	// Once set, it can not be changed. This field will be wiped when a service is updated to a non 'LoadBalancer' type.
	// +featureGate=LoadBalancerClass
	// +optional
	LoadBalancerClass *string `json:"loadBalancerClass,omitempty" protobuf:"bytes,21,opt,name=loadBalancerClass"`

	// InternalTrafficPolicy specifies if the cluster internal traffic
	// should be routed to all endpoints or node-local endpoints only.
	// "Cluster" routes internal traffic to a Service to all endpoints.
	// "Local" routes traffic to node-local endpoints only, traffic is
	// dropped if no node-local endpoints are ready.
	// The default value is "Cluster".
	// +featureGate=ServiceInternalTrafficPolicy
	// +optional
	InternalTrafficPolicy *ServiceInternalTrafficPolicyType `json:"internalTrafficPolicy,omitempty" protobuf:"bytes,22,opt,name=internalTrafficPolicy"`
}

type LoadBalancerStatus struct {
        // ingress-nginx是组路由规则，这里是路由规则列表
        Ingress []LoadBalancerIngress `json:"ingress,omitempty" protobuf:"bytes,1,rep,name=ingress"`
} 

type ServiceStatus struct {
        // ingress 状态
        LoadBalancer LoadBalancerStatus `json:"loadBalancer,omitempty" protobuf:"bytes,1,opt,name=loadBalancer"`
        // Current service state
        // +optional
        // +patchMergeKey=type
        // +patchStrategy=merge
        // +listType=map
        // +listMapKey=type
        Conditions []metav1.Condition `json:"conditions,omitempty" patchStrategy:"merge" patchMergeKey:"type" protobuf:"bytes,2,rep,name=conditions"`
}

type LoadBalancerIngress struct {
        // IP is set for load-balancer ingress points that are IP based
        // (typically GCE or OpenStack load-balancers)
        // +optional
        IP string `json:"ip,omitempty" protobuf:"bytes,1,opt,name=ip"`

        // Hostname is set for load-balancer ingress points that are DNS based
        // (typically AWS load-balancers)
        // +optional
        Hostname string `json:"hostname,omitempty" protobuf:"bytes,2,opt,name=hostname"`

        // Ports is a list of records of service ports
        // If used, every port defined in the service should have an entry in it
        // +listType=atomic
        // +optional
        Ports []PortStatus `json:"ports,omitempty" protobuf:"bytes,4,rep,name=ports"`
}

type PortStatus struct {
        // Port is the port number of the service port of which status is recorded here
        Port int32 `json:"port" protobuf:"varint,1,opt,name=port"`
        
        // Protocol is the protocol of the service port of which status is recorded here
        // The supported values are: "TCP", "UDP", "SCTP"
        Protocol Protocol `json:"protocol" protobuf:"bytes,2,opt,name=protocol,casttype=Protocol"`

        Error *string `json:"error,omitempty" protobuf:"bytes,3,opt,name=error"`
}

4 service/ingress暴露端口映射关系：

5 挂载容器运行时：

“/var/run/netns”, “/run/containerd”, “/run/xtables.lock” 是三个linux系统内核特殊目录，用于与内核虚拟化接口进行交互

6 port、nodeport、targetport的关系如下：

• port: 服务(被k8s集群以外的组件)访问的端口
• targetPort: 容器内部通讯端口
• nodePort:: k8s外部的组件调用pod的一种方式叫NodePort，另外一种方式是LoadBalaner