K8s TLS Bootstrap机制

流程图：

场景：

当集群开启了 TLS 认证后，每个节点的 kubelet 组件都要使用由 apiserver 使用的 CA 签发的有效证书才能与 apiserver 通讯；此时如果节点多起来，为每个节点单独签署证书将是一件非常繁琐的事情；TLS bootstrapping 功能就是让 kubelet 先使用一个预定的低权限用户连接到 apiserver，然后向 apiserver 申请证书，kubelet 的证书由 apiserver 动态签署

TLS BootStrap机制与RBAC机制

TLS：

TLS 用来对通讯加密，防止中间人窃听，如果证书不信任就无法与 apiserver 建议连接，更不用提有没有权限向 api 请求指定内容。

RBAC：

RBAC 模型 在 TLS 解决了鉴权问题，RBAC 会规定用户或者用户组具有请求那些 aip 的权限，配合 TLS 加密后，就能对发起的操作进行认证与鉴权。

缺一不可：

缺少TLS：通讯过程不安全，内容会被窃听
缺少RBAC：对操作过程不安全，虽然token能用来加密，解决窃听，但是如果不限制这个token的权限，就有可能这个token被用于其它非法目的，或者由于误操作，但是没鉴权，导致不必要的后果发生

TLS与RBAC配合：

当节点首次请求时，kubelet 使用 bootstrap.kubeconfig 中 apiserver 的 CA 证书与 appserver 建立 TLS 连接，同时还需要使用 bootstrap.kubeconfig 中的 用户 token 来向 apiserver 证明自己的 RBAC 授权身份。

kubelet如何使用TLS BootStrap证书

既然 TLS bootstrapping 功能是让 kubelet 组件去 apiserver 申请证书，然后用于连接 apiserver；那么第一次启动时没有证书如何连接 apiserver ?

当您运行 kubeadm join 时：

1、kubeadm 使用 Bootstrap Token 凭证来执行 TLS 引导，它获取下载 kubelet-config ConfigMap 所需的凭证并将其写入 /var/lib/kubelet/config.yaml。
即：节点kubelet的配置是kubeadm通过下载 kubelet-config ConfigMap来获取内容，并写入到/var/lib/kubelet/config.yaml

kubelet-config configmap example：

apiVersion: v1
data:
  kubelet: |
    apiVersion: kubelet.config.k8s.io/v1beta1
    authentication:
      anonymous:
        enabled: false
      webhook:
        cacheTTL: 0s
        enabled: true
      x509:
        clientCAFile: /etc/kubernetes/pki/ca.crt
    authorization:
      mode: Webhook
      webhook:
        cacheAuthorizedTTL: 0s
        cacheUnauthorizedTTL: 0s
    cgroupDriver: cgroupfs
    clusterDNS:
    - 10.96.0.10
    clusterDomain: cluster.local
    cpuManagerReconcilePeriod: 0s
    evictionPressureTransitionPeriod: 0s
    fileCheckFrequency: 0s
    healthzBindAddress: 127.0.0.1
    healthzPort: 10248
    httpCheckFrequency: 0s
    imageMinimumGCAge: 0s
    kind: KubeletConfiguration
    logging: {}
    nodeStatusReportFrequency: 0s
    nodeStatusUpdateFrequency: 0s
    resolvConf: /run/systemd/resolve/resolv.conf
    rotateCertificates: true
    runtimeRequestTimeout: 0s
    shutdownGracePeriod: 0s
    shutdownGracePeriodCriticalPods: 0s
    staticPodPath: /etc/kubernetes/manifests
    streamingConnectionIdleTimeout: 0s
    syncFrequency: 0s
    volumeStatsAggPeriod: 0s
kind: ConfigMap
metadata:
  annotations:
    kubeadm.kubernetes.io/component-config.hash: sha256:306a726156f1e2879bedabbdfa452caae8a63929426a55de71c22fe901fde977
  name: kubelet-config-1.20
  namespace: kube-system

2、kubeadm 运行以下两个命令将新配置加载到 kubelet 中，并启动kubelet：
systemctl daemon-reload && systemctl restart kubelet
3、在 kubelet 加载新配置后，kubeadm 将写入 /etc/kubernetes/bootstrap-kubelet.conf KubeConfig 文件中， 该文件包含 CA 证书和引导程序令牌（token）
4、kubelet看到它没有kubeconfig文件
5、kubelet搜索并查找bootstrap-kubeconfig文件
6、kubelet读取它的bootstrap文件，检索API server的URL和一个低权限的“token”
7、kubelet连接到API服务器，使用token进行身份验证
8、kubelet现在具有创建和检索证书签名请求（CSR）的有限凭据
9、kubelet为自己创建了一个CSR
10、CSR通过以下两种方式之一获得批准：
如果已配置，kube-controller-manager将自动批准CSR
如果已配置，则外部流程（可能是人员）使用Kubernetes API或通过批准CSR kubectl
、为kubelet创建证书
11、证书颁发给kubelet
12、kubelet检索证书
13、kubelet 使用密钥和签名证书创建一个正确的kubeconfig文件。kubelet 使用这些证书执行 TLS 引导程序并获取唯一的凭据，该凭据被存储在 /etc/kubernetes/kubelet.conf 中。
14、kubelet开始正常运作
15、当 /etc/kubernetes/kubelet.conf 文件被写入后，kubelet 就完成了 TLS 引导过程。 Kubeadm 在完成 TLS 引导过程后将删除 /etc/kubernetes/bootstrap-kubelet.conf 文件。
16、可选：如果已配置，则当证书接近到期时，kubelet会自动请求更新证书

进入主题之：TLS Bootstrap机制在kubeadm init需要做的事

1、如果没有指定token，那么生成默认的default bootstrap token，就是kubeadm init后print的那个

// DefaultedInitConfiguration takes a versioned init config (often populated by flags), defaults it and converts it into internal InitConfiguration
func DefaultedInitConfiguration(versionedInitCfg *kubeadmapiv1beta2.InitConfiguration, versionedClusterCfg *kubeadmapiv1beta2.ClusterConfiguration) (*kubeadmapi.InitConfiguration, error) {
	internalcfg := &kubeadmapi.InitConfiguration{}

	// Takes passed flags into account; the defaulting is executed once again enforcing assignment of
	// static default values to cfg only for values not provided with flags
	kubeadmscheme.Scheme.Default(versionedInitCfg)
	if err := kubeadmscheme.Scheme.Convert(versionedInitCfg, internalcfg, nil); err != nil {
		return nil, err
	}

	kubeadmscheme.Scheme.Default(versionedClusterCfg)
	if err := kubeadmscheme.Scheme.Convert(versionedClusterCfg, &internalcfg.ClusterConfiguration, nil); err != nil {
		return nil, err
	}

	// Applies dynamic defaults to settings not provided with flags
	// 生成默认的一些配置，比如：default bootstrap token
	if err := SetInitDynamicDefaults(internalcfg); err != nil {
		return nil, err
	}
	// Validates cfg (flags/configs + defaults + dynamic defaults)
	if err := validation.ValidateInitConfiguration(internalcfg).ToAggregate(); err != nil {
		return nil, err
	}
	return internalcfg, nil
}

// SetInitDynamicDefaults checks and sets configuration values for the InitConfiguration object
func SetInitDynamicDefaults(cfg *kubeadmapi.InitConfiguration) error {
    // 生成default bootstrap token，如果没有kubeadm init的时候没有指定token到话
	if err := SetBootstrapTokensDynamicDefaults(&cfg.BootstrapTokens); err != nil {
		return err
	}
	if err := SetNodeRegistrationDynamicDefaults(&cfg.NodeRegistration, true); err != nil {
		return err
	}
	if err := SetAPIEndpointDynamicDefaults(&cfg.LocalAPIEndpoint); err != nil {
		return err
	}
	return SetClusterDynamicDefaults(&cfg.ClusterConfiguration, &cfg.LocalAPIEndpoint, &cfg.NodeRegistration)
}

// 生成一个随机的default bootstrap token
func SetBootstrapTokensDynamicDefaults(cfg *[]kubeadmapi.BootstrapToken) error {
	// Populate the .Token field with a random value if unset
	// We do this at this layer, and not the API defaulting layer
	// because of possible security concerns, and more practically
	// because we can't return errors in the API object defaulting
	// process but here we can.
	for i, bt := range *cfg {
		if bt.Token != nil && len(bt.Token.String()) > 0 {
			continue
		}

		tokenStr, err := bootstraputil.GenerateBootstrapToken()
		if err != nil {
			return errors.Wrap(err, "couldn't generate random token")
		}
		token, err := kubeadmapi.NewBootstrapTokenString(tokenStr)
		if err != nil {
			return err
		}
		(*cfg)[i].Token = token
	}

	return nil
}

2、为这个token生成相关联的RBAC对象

func runBootstrapToken(c workflow.RunData) error {
	data, ok := c.(InitData)
	if !ok {
		return errors.New("bootstrap-token phase invoked with an invalid data struct")
	}

	client, err := data.Client()
	if err != nil {
		return err
	}

	if !data.SkipTokenPrint() {
		tokens := data.Tokens()
		if len(tokens) == 1 {
			fmt.Printf("[bootstrap-token] Using token: %s\n", tokens[0])
		} else if len(tokens) > 1 {
			fmt.Printf("[bootstrap-token] Using tokens: %v\n", tokens)
		}
	}

	fmt.Println("[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles")
	// Create the default node bootstrap token
	if err := nodebootstraptokenphase.UpdateOrCreateTokens(client, false, data.Cfg().BootstrapTokens); err != nil {
		return errors.Wrap(err, "error updating or creating token")
	}
    // 创建与token相关联的RBAC对象
	// Create RBAC rules that makes the bootstrap tokens able to get nodes
	if err := nodebootstraptokenphase.AllowBoostrapTokensToGetNodes(client); err != nil {
		return errors.Wrap(err, "error allowing bootstrap tokens to get Nodes")
	}
	// Create RBAC rules that makes the bootstrap tokens able to post CSRs
	if err := nodebootstraptokenphase.AllowBootstrapTokensToPostCSRs(client); err != nil {
		return errors.Wrap(err, "error allowing bootstrap tokens to post CSRs")
	}
	// Create RBAC rules that makes the bootstrap tokens able to get their CSRs approved automatically
	if err := nodebootstraptokenphase.AutoApproveNodeBootstrapTokens(client); err != nil {
		return errors.Wrap(err, "error auto-approving node bootstrap tokens")
	}

	// Create/update RBAC rules that makes the nodes to rotate certificates and get their CSRs approved automatically
	if err := nodebootstraptokenphase.AutoApproveNodeCertificateRotation(client); err != nil {
		return err
	}

	// Create the cluster-info ConfigMap with the associated RBAC rules
	if err := clusterinfophase.CreateBootstrapConfigMapIfNotExists(client, data.KubeConfigPath()); err != nil {
		return errors.Wrap(err, "error creating bootstrap ConfigMap")
	}
	if err := clusterinfophase.CreateClusterInfoRBACRules(client); err != nil {
		return errors.Wrap(err, "error creating clusterinfo RBAC rules")
	}
	return nil
}

3、生成cluster-info configmap以及对应的RBAC对象，token配合cluster-info这个configmap里面的url，ca证书等信息，组成可以访问apiserver的请求，从而完成后面的证书申请和轮换

这里的RBAC对象是为了让未授权的人也能读取cluster-info这个configmap

func runBootstrapToken(c workflow.RunData) error {
	data, ok := c.(InitData)
	if !ok {
		return errors.New("bootstrap-token phase invoked with an invalid data struct")
	}

	client, err := data.Client()
	if err != nil {
		return err
	}

	if !data.SkipTokenPrint() {
		tokens := data.Tokens()
		if len(tokens) == 1 {
			fmt.Printf("[bootstrap-token] Using token: %s\n", tokens[0])
		} else if len(tokens) > 1 {
			fmt.Printf("[bootstrap-token] Using tokens: %v\n", tokens)
		}
	}

	fmt.Println("[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles")
	// Create the default node bootstrap token
	if err := nodebootstraptokenphase.UpdateOrCreateTokens(client, false, data.Cfg().BootstrapTokens); err != nil {
		return errors.Wrap(err, "error updating or creating token")
	}
	// Create RBAC rules that makes the bootstrap tokens able to get nodes
	if err := nodebootstraptokenphase.AllowBoostrapTokensToGetNodes(client); err != nil {
		return errors.Wrap(err, "error allowing bootstrap tokens to get Nodes")
	}
	// Create RBAC rules that makes the bootstrap tokens able to post CSRs
	if err := nodebootstraptokenphase.AllowBootstrapTokensToPostCSRs(client); err != nil {
		return errors.Wrap(err, "error allowing bootstrap tokens to post CSRs")
	}
	// Create RBAC rules that makes the bootstrap tokens able to get their CSRs approved automatically
	if err := nodebootstraptokenphase.AutoApproveNodeBootstrapTokens(client); err != nil {
		return errors.Wrap(err, "error auto-approving node bootstrap tokens")
	}

	// Create/update RBAC rules that makes the nodes to rotate certificates and get their CSRs approved automatically
	if err := nodebootstraptokenphase.AutoApproveNodeCertificateRotation(client); err != nil {
		return err
	}

    // 创建cluster-info configmap
	// Create the cluster-info ConfigMap with the associated RBAC rules
	if err := clusterinfophase.CreateBootstrapConfigMapIfNotExists(client, data.KubeConfigPath()); err != nil {
		return errors.Wrap(err, "error creating bootstrap ConfigMap")
	}
	if err := clusterinfophase.CreateClusterInfoRBACRules(client); err != nil {
		return errors.Wrap(err, "error creating clusterinfo RBAC rules")
	}
	return nil
}

创建专门的RBAC对象，使得未认证的用户也能读取cluster-info这个configmap

// CreateClusterInfoRBACRules creates the RBAC rules for exposing the cluster-info ConfigMap in the kube-public namespace to unauthenticated users
func CreateClusterInfoRBACRules(client clientset.Interface) error {
	klog.V(1).Infoln("creating the RBAC rules for exposing the cluster-info ConfigMap in the kube-public namespace")
	err := apiclient.CreateOrUpdateRole(client, &rbac.Role{
		ObjectMeta: metav1.ObjectMeta{
			Name:      BootstrapSignerClusterRoleName,
			Namespace: metav1.NamespacePublic,
		},
		Rules: []rbac.PolicyRule{
			{
				Verbs:         []string{"get"},
				APIGroups:     []string{""},
				Resources:     []string{"configmaps"},
				ResourceNames: []string{bootstrapapi.ConfigMapClusterInfo},
			},
		},
	})
	if err != nil {
		return err
	}

	return apiclient.CreateOrUpdateRoleBinding(client, &rbac.RoleBinding{
		ObjectMeta: metav1.ObjectMeta{
			Name:      BootstrapSignerClusterRoleName,
			Namespace: metav1.NamespacePublic,
		},
		RoleRef: rbac.RoleRef{
			APIGroup: rbac.GroupName,
			Kind:     "Role",
			Name:     BootstrapSignerClusterRoleName,
		},
		Subjects: []rbac.Subject{
			{
				Kind: rbac.UserKind,
				// 这里指定了任何用户都可以读取cluster-info这个configmap，因为刚开始join的时候，
				// 就是要从这里面才能读取到集群的ca证书，apiserver url从而知道master信息并认证maste身份
				Name: user.Anonymous,
			},
		},
	})
}

进入主题之：TLS Bootstrap机制在kubeadm node join需要做的事

1、从cluster-info中获取集群信息和用于集群身份认证的证书

func getKubeletStartJoinData(c workflow.RunData) (*kubeadmapi.JoinConfiguration, *kubeadmapi.InitConfiguration, *clientcmdapi.Config, error) {
	data, ok := c.(JoinData)
	if !ok {
		return nil, nil, nil, errors.New("kubelet-start phase invoked with an invalid data struct")
	}
	cfg := data.Cfg()
	initCfg, err := data.InitCfg()
	if err != nil {
		return nil, nil, nil, err
	}
	// 生成tlsBootstrapCfg
	tlsBootstrapCfg, err := data.TLSBootstrapCfg()
	if err != nil {
		return nil, nil, nil, err
	}
	return cfg, initCfg, tlsBootstrapCfg, nil
}

// TLSBootstrapCfg returns the cluster-info (kubeconfig).
func (j *joinData) TLSBootstrapCfg() (*clientcmdapi.Config, error) {
	if j.tlsBootstrapCfg != nil {
		return j.tlsBootstrapCfg, nil
	}
	klog.V(1).Infoln("[preflight] Discovering cluster-info")
	// 读取cluster-info configmap的内容，同时配合join token生成tlsBootstrapCfg
	tlsBootstrapCfg, err := discovery.For(j.cfg)
	j.tlsBootstrapCfg = tlsBootstrapCfg
	return tlsBootstrapCfg, err
}

// For returns a kubeconfig object that can be used for doing the TLS Bootstrap with the right credentials
// Also, before returning anything, it makes sure it can trust the API Server
// 读取cluster-info configmap的内容，同时配合join token生成tlsBootstrapCfg
func For(cfg *kubeadmapi.JoinConfiguration) (*clientcmdapi.Config, error) {
	// TODO: Print summary info about the CA certificate, along with the checksum signature
	// we also need an ability for the user to configure the client to validate received CA cert against a checksum
	config, err := DiscoverValidatedKubeConfig(cfg)
	if err != nil {
		return nil, errors.Wrap(err, "couldn't validate the identity of the API Server")
	}

	// If the users has provided a TLSBootstrapToken use it for the join process.
	// This is usually the case of Token discovery, but it can also be used with a discovery file
	// without embedded authentication credentials.
	if len(cfg.Discovery.TLSBootstrapToken) != 0 {
		klog.V(1).Info("[discovery] Using provided TLSBootstrapToken as authentication credentials for the join process")
        // 从cluster-info configmap读取的内容
		clusterinfo := kubeconfigutil.GetClusterFromKubeConfig(config)
		// 使用从cluster-info configmap读取的apiserver信息，加上我们的token组成了bootstrap config
		return kubeconfigutil.CreateWithToken(
			clusterinfo.Server,
			kubeadmapiv1beta2.DefaultClusterName,
			TokenUser,
			clusterinfo.CertificateAuthorityData,
			cfg.Discovery.TLSBootstrapToken,
		), nil
	}
    ...
    ...
}

// DiscoverValidatedKubeConfig returns a validated Config object that specifies where the cluster is and the CA cert to trust
// 读取cluster-info configmap
func DiscoverValidatedKubeConfig(cfg *kubeadmapi.JoinConfiguration) (*clientcmdapi.Config, error) {
	switch {
	case cfg.Discovery.File != nil:
		kubeConfigPath := cfg.Discovery.File.KubeConfigPath
		if isHTTPSURL(kubeConfigPath) {
			return https.RetrieveValidatedConfigInfo(kubeConfigPath, kubeadmapiv1beta2.DefaultClusterName, cfg.Discovery.Timeout.Duration)
		}
		return file.RetrieveValidatedConfigInfo(kubeConfigPath, kubeadmapiv1beta2.DefaultClusterName, cfg.Discovery.Timeout.Duration)
	case cfg.Discovery.BootstrapToken != nil:
		return token.RetrieveValidatedConfigInfo(&cfg.Discovery)
	default:
		return nil, errors.New("couldn't find a valid discovery configuration")
	}
}

func RetrieveValidatedConfigInfo(cfg *kubeadmapi.Discovery) (*clientcmdapi.Config, error) {
	return retrieveValidatedConfigInfo(nil, cfg, constants.DiscoveryRetryInterval)
}

// retrieveValidatedConfigInfo is a private implementation of RetrieveValidatedConfigInfo.
// It accepts an optional clientset that can be used for testing purposes.
func retrieveValidatedConfigInfo(client clientset.Interface, cfg *kubeadmapi.Discovery, interval time.Duration) (*clientcmdapi.Config, error) {
	token, err := kubeadmapi.NewBootstrapTokenString(cfg.BootstrapToken.Token)
	if err != nil {
		return nil, err
	}

	// Load the CACertHashes into a pubkeypin.Set
	pubKeyPins := pubkeypin.NewSet()
	if err = pubKeyPins.Allow(cfg.BootstrapToken.CACertHashes...); err != nil {
		return nil, err
	}

	duration := cfg.Timeout.Duration
	// Make sure the interval is not bigger than the duration
	if interval > duration {
		interval = duration
	}

	endpoint := cfg.BootstrapToken.APIServerEndpoint
	insecureBootstrapConfig := buildInsecureBootstrapKubeConfig(endpoint, kubeadmapiv1beta2.DefaultClusterName)
	clusterName := insecureBootstrapConfig.Contexts[insecureBootstrapConfig.CurrentContext].Cluster

	klog.V(1).Infof("[discovery] Created cluster-info discovery client, requesting info from %q", endpoint)
	// 从集群中获取cluster-info这个configmap的内容
	insecureClusterInfo, err := getClusterInfo(client, insecureBootstrapConfig, token, interval, duration)
    ...
    ...
}

func getClusterInfo(client clientset.Interface, kubeconfig *clientcmdapi.Config, token *kubeadmapi.BootstrapTokenString, interval, duration time.Duration) (*v1.ConfigMap, error) {
	var cm *v1.ConfigMap
	var err error

	// Create client from kubeconfig
	if client == nil {
		client, err = kubeconfigutil.ToClientSet(kubeconfig)
		if err != nil {
			return nil, err
		}
	}

	ctx, cancel := context.WithTimeout(context.TODO(), duration)
	defer cancel()

	wait.JitterUntil(func() {
	    // 获取cluster-info这个configmap
		cm, err = client.CoreV1().ConfigMaps(metav1.NamespacePublic).Get(context.TODO(), bootstrapapi.ConfigMapClusterInfo, metav1.GetOptions{})
		if err != nil {
			klog.V(1).Infof("[discovery] Failed to request cluster-info, will try again: %v", err)
			return
		}
		// Even if the ConfigMap is available the JWS signature is patched-in a bit later.
		// Make sure we retry util then.
		if _, ok := cm.Data[bootstrapapi.JWSSignatureKeyPrefix+token.ID]; !ok {
			klog.V(1).Infof("[discovery] The cluster-info ConfigMap does not yet contain a JWS signature for token ID %q, will try again", token.ID)
			err = errors.Errorf("could not find a JWS signature in the cluster-info ConfigMap for token ID %q", token.ID)
			return
		}
		// Cancel the context on success
		cancel()
	}, interval, 0.3, true, ctx.Done())

	if err != nil {
		return nil, err
	}

	return cm, nil
}

2、使用token配合从cluster-info中获取的内容，生成bootstrap-kubelet.conf

// runKubeletStartJoinPhase executes the kubelet TLS bootstrap process.
// This process is executed by the kubelet and completes with the node joining the cluster
// with a dedicates set of credentials as required by the node authorizer
func runKubeletStartJoinPhase(c workflow.RunData) (returnErr error) {
	cfg, initCfg, tlsBootstrapCfg, err := getKubeletStartJoinData(c)
	if err != nil {
		return err
	}
	bootstrapKubeConfigFile := kubeadmconstants.GetBootstrapKubeletKubeConfigPath()

	// Deletes the bootstrapKubeConfigFile, so the credential used for TLS bootstrap is removed from disk
	defer os.Remove(bootstrapKubeConfigFile)

	// Write the bootstrap kubelet config file or the TLS-Bootstrapped kubelet config file down to disk
	// 将tlsBootstrapCfg的内容写入到本地磁盘的bootstrap-kubelet.conf
	klog.V(1).Infof("[kubelet-start] writing bootstrap kubelet config file at %s", bootstrapKubeConfigFile)
	if err := kubeconfigutil.WriteToDisk(bootstrapKubeConfigFile, tlsBootstrapCfg); err != nil {
		return errors.Wrap(err, "couldn't save bootstrap-kubelet.conf to disk")
	}
	...
	...
}

// GetBootstrapKubeletKubeConfigPath returns the location on the disk where bootstrap kubelet kubeconfig is located by default
func GetBootstrapKubeletKubeConfigPath() string {
	return filepath.Join(KubernetesDir, KubeletBootstrapKubeConfigFileName)
}

const KubeletBootstrapKubeConfigFileName untyped string = "bootstrap-kubelet.conf"

3、启动kubelet

// runKubeletStartJoinPhase executes the kubelet TLS bootstrap process.
// This process is executed by the kubelet and completes with the node joining the cluster
// with a dedicates set of credentials as required by the node authorizer
func runKubeletStartJoinPhase(c workflow.RunData) (returnErr error) {
    ...
    ...
	// Try to start the kubelet service in case it's inactive
	fmt.Println("[kubelet-start] Starting the kubelet")
	kubeletphase.TryStartKubelet()
	...
}

// TryStartKubelet attempts to bring up kubelet service
func TryStartKubelet() {
	// If we notice that the kubelet service is inactive, try to start it
	initSystem, err := initsystem.GetInitSystem()
	if err != nil {
		fmt.Println("[kubelet-start] no supported init system detected, won't make sure the kubelet is running properly.")
		return
	}

	if !initSystem.ServiceExists("kubelet") {
		fmt.Println("[kubelet-start] couldn't detect a kubelet service, can't make sure the kubelet is running properly.")
	}

	// This runs "systemctl daemon-reload && systemctl restart kubelet"
	if err := initSystem.ServiceRestart("kubelet"); err != nil {
		fmt.Printf("[kubelet-start] WARNING: unable to start the kubelet service: [%v]\n", err)
		fmt.Printf("[kubelet-start] Please ensure kubelet is reloaded and running manually.\n")
	}
}

4、kubelet会使用bootstrap-kubelet.conf去做新证书申请，然后轮换，生成/etc/kubernetes/kubelet.conf

5、后续kubeadm在kubelet完成证书轮换，生成新的/etc/kubernetes/kubelet.conf后删除bootstrap-kubelet.conf

// runKubeletStartJoinPhase executes the kubelet TLS bootstrap process.
// This process is executed by the kubelet and completes with the node joining the cluster
// with a dedicates set of credentials as required by the node authorizer
func runKubeletStartJoinPhase(c workflow.RunData) (returnErr error) {
	cfg, initCfg, tlsBootstrapCfg, err := getKubeletStartJoinData(c)
	if err != nil {
		return err
	}
	bootstrapKubeConfigFile := kubeadmconstants.GetBootstrapKubeletKubeConfigPath()

	// Deletes the bootstrapKubeConfigFile, so the credential used for TLS bootstrap is removed from disk
	// kubelet完成证书轮换后，最后kubeadm会删除bootstrap-kubelet.conf
	defer os.Remove(bootstrapKubeConfigFile)
    ...
    ...

综合剖析篇：Bootstrap token如何与RBAC机制结合进行认证

1、token对应的secret和RBAC对象分析

1、生成bootstrap token，创建bootstrap token secret；

apiVersion: v1
data:
  // 其代表的用户所在用户组为system:bootstrappers:kubeadm:default-node-token；
  auth-extra-groups: system:bootstrappers:kubeadm:default-node-token
  expiration: 2022-04-03T11:13:09+08:00
  token-id: abcdef
  token-secret: 0123456789abcdef
  usage-bootstrap-authentication: "true"
  usage-bootstrap-signing: "true"
kind: Secret
metadata:
  name: bootstrap-token-abcdef
  namespace: kube-system
type: bootstrap.kubernetes.io/token

关于bootstrap token secret相关的格式说明：

secret的name格式为bootstrap-token-{token-id}的格式；

secret的type固定为bootstrap.kubernetes.io/token；

secret data中的token-id为6位数字字母组合字符串，token-secret为16位数字字母组合字符串；

secret data中的auth-extra-groups定义了bootstrap token所代表用户所属的的group，
kubeadm使用了system:bootstrappers:kubeadm:default-node-token；

secret所对应的bootstrap token为{token-id}.{token-secret}；

2、授予bootstrap token创建CSR证书签名请求的权限，即授予kubelet创建CSR证书签名请求的权限；

即创建ClusterRoleBinding – kubeadm:kubelet-bootstrap

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubeadm:kubelet-bootstrap
  ...
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node-bootstrapper
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:kubeadm:default-node-token

kubeadm生成的bootstrap token所代表的用户所在用户组为system:bootstrappers:kubeadm:default-node-token，所以这里绑定权限的时候将权限绑定给了用户组system:bootstrappers:kubeadm:default-node-token；

接下来看下被授予的权限ClusterRole – system:node-bootstrapper

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:node-bootstrapper
  ...
rules:
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests
  verbs:
  - create
  - get
  - list
  - watch

3、授予bootstrap token权限，让kube-controller-manager可以自动审批其发起的CSR；

即创建ClusterRoleBinding – kubeadm:node-autoapprove-bootstrap

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubeadm:node-autoapprove-bootstrap
  ...
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:kubeadm:default-node-token

kubeadm生成的bootstrap token所代表的用户所在用户组为system:bootstrappers:kubeadm:default-node-token，所以这里绑定权限的时候将权限绑定给了用户组system:bootstrappers:kubeadm:default-node-token；

接下来看下被授予的权限ClusterRole – system:certificates.k8s.io:certificatesigningrequests:nodeclient

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
  ...
rules:
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests/nodeclient
  verbs:
  - create

4、授予kubelet权限，让kube-controller-manager自动批复kubelet的证书轮换请求；

即创建ClusterRoleBinding – kubeadm:node-autoapprove-certificate-rotation

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubeadm:node-autoapprove-certificate-rotation
  ...
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:nodes

kubelet创建的CSR用户名格式为system:node:，用户组为system:nodes，所以kube-controller-manager为kubelet生成的证书所代表的用户所在用户组为system:nodes，所以这里绑定权限的时候将权限绑定给了用户组system:nodes；

接下来看下被授予的权限ClusterRole – system:certificates.k8s.io:certificatesigningrequests:selfnodeclient

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
  ...
rules:
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests/selfnodeclient
  verbs:
  - create

2、token本身的校验（鉴权由RBAC负责，此处是校验token是否合法）

假设：token为c8ad9c.2e4d610cf3e7426e（格式为：tokenID.tokenSecret）

apiserver根据tokenID，加上bootstrap-token前缀，找到对应的secret

kubectl get secret -n kube-system

NAME                TYPE DATA AGE
bootstrap-token-c8ad9c bootstrap.kubernetes.io/token 6 3d18h

读取secret中的内容，得到token-id，token-secret

kubectl get secret -n kube-system bootstrap-token-c8ad9c -n kube-system -oyaml

apiVersion: v1
data:
auth-extra-groups: c3lzdGVtOmJvb3RzdHJhcHBlcnM6ZGVmYXVsdC1ub2RlLXRva2VuLHN5c3RlbTpib290c3RyYXBwZXJzOndvcmtlcixzeXN0ZW06Ym9vdHN0cmFwcGVyczppbmdyZXNz
description: VGhlIGRlZmF1bHQgYm9vdHN0cmFwIHRva2VuIGdlbmVyYXRlZCBieSAna3ViZWxldCAnLg==
token-id: YzhhZDlj
token-secret: MmU0ZDYxMGNmM2U3NDI2ZQ==
usage-bootstrap-authentication: dHJ1ZQ==
usage-bootstrap-signing: dHJ1ZQ==
kind: Secret
metadata:
name: bootstrap-token-c8ad9c
namespace: kube-system
type: bootstrap.kubernetes.io/token

解码token-id与token-secret，组成tokenID.tokenSecret形式，看看是否等于我们的join token

解密token-id：
echo "YzhhZDlj" | base64 -d
c8ad9c

解密token-secret：
echo "MmU0ZDYxMGNmM2U3NDI2ZQ==" | base64 -d
2e4d610cf3e7426e

组成c8ad9c.2e4d610cf3e7426e，与我们的join token的值c8ad9c.2e4d610cf3e7426e一致，因此认证通过，认为token有效

使用此token可以获得的权限就是auth-extra-groups这个组具有的权限，这个组的权限就是上面RBAC对象赋予的

解析auth-extra-groups：

echo "c3lzdGVtOmJvb3RzdHJhcHBlcnM6ZGVmYXVsdC1ub2RlLXRva2VuLHN5c3RlbTpib290c3RyYXBwZXJzOndvcmtlcixzeXN0ZW06Ym9vdHN0cmFwcGVyczppbmdyZXNz" | base64 -d

system:bootstrappers:default-node-token,system:bootstrappers:worker,system:bootstrappers:ingress

查看对应的RBAC对象

1、kubectl get clusterrole

NAME CREATED AT
system:node-bootstrapper 2021-07-09T09:34:55Z

2、kubectl get clusterrole system:node-bootstrapper -o yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:node-bootstrapper
rules:
- apiGroups:
- certificates.k8s.io
resources:
- certificatesigningrequests # 定义了一个csr权限
verbs:
- create
- get
- list
- watch


3、kubectl get clusterrolebinding kubelet-bootstrap -o yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubelet-bootstrap

roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:node-bootstrapper
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:bootstrappers:default-node-token

可以看到clusterrolebinding把一个名为system:node-bootstrapper的ClusterRole绑定到一个名为system:bootstrappers:default-node-token的Group，使得这个Group具有system:node-bootstrapper这个ClusterRole所指定的权限

附篇：kubeadm master join时不依赖TLS Bootstrap机制

kubeadm join example：

// node join
kubeadm join 192.168.1.10:6443
 --token 42ojpt.z2h5ii9n898tzo36 
 --discovery-token-ca-cert-hash sha256:7cf14e8cb965d5eb9d66f3707ba20deeadc90bd36b730ce4c0e5d9db80d3625b

// master join
kubeadm join 192.168.1.10:6443
 --token 42ojpt.z2h5ii9n898tzo36
 --discovery-token-ca-cert-hash sha256:7cf14e8cb965d5eb9d66f3707ba20deeadc90bd36b730ce4c0e5d9db80d3625b
 --certificate-key e799a655f667fc327ab8c91f4f2541b57b96d2693ab5af96314ebddea7a68526
 --experimental-control-plane 

master join与 node join区别：

master join的时候，多传了一个certificate-key，这个是为了解密kubeadm-cert这个secret里面的内容，以从中获取控制面证书，因为master节点不能自己签自己，不能自己给自己鉴权，所以将证书信息放到了kubeadm-cert这个secret中。
所以certificate-key是kubeadm-cert secret中用于加密control-plane证书的key。

master join不依赖TLS Bootstrap机制，那么如何实现认证与鉴权呢？实现机制是什么？

实现机制依赖的步骤一：kubeadm init阶段将控制面证书用key加密后保存到集群供后续masrer解密后获取

源码剖析：

// master init阶段的UploadCerts阶段
// 负责将控制面证书用key加密后保存到集群中的kubeadm-cert secret中
func runUploadCerts(c workflow.RunData) error {
	data, ok := c.(InitData)
	if !ok {
		return errors.New("upload-certs phase invoked with an invalid data struct")
	}

	if !data.UploadCerts() {
		fmt.Printf("[upload-certs] Skipping phase. Please see --%s\n", options.UploadCerts)
		return nil
	}
	client, err := data.Client()
	if err != nil {
		return err
	}

	if len(data.CertificateKey()) == 0 {
		certificateKey, err := copycerts.CreateCertificateKey()
		if err != nil {
			return err
		}
		data.SetCertificateKey(certificateKey)
	}
    // 上传证书到集群
	if err := copycerts.UploadCerts(client, data.Cfg(), data.CertificateKey()); err != nil {
		return errors.Wrap(err, "error uploading certs")
	}
	if !data.SkipCertificateKeyPrint() {
		fmt.Printf("[upload-certs] Using certificate key:\n%s\n", data.CertificateKey())
	}
	return nil
}

// UploadCerts save certs needs to join a new control-plane on kubeadm-certs sercret.
// 负责生成kubeadm-certs sercret，并上传
func UploadCerts(client clientset.Interface, cfg *kubeadmapi.InitConfiguration, key string) error {
	fmt.Printf("[upload-certs] Storing the certificates in Secret %q in the %q Namespace\n", kubeadmconstants.KubeadmCertsSecret, metav1.NamespaceSystem)
	// 对加密的key进行编码
	decodedKey, err := hex.DecodeString(key)
	if err != nil {
		return errors.Wrap(err, "error decoding certificate key")
	}
	tokenID, err := createShortLivedBootstrapToken(client)
	if err != nil {
		return err
	}
    // 从磁盘上获取控制面证书，并使用key进行加密，这部分就是kubeadm-cert这个secret的内容
	secretData, err := getDataFromDisk(cfg, decodedKey)
	if err != nil {
		return err
	}
	ref, err := getSecretOwnerRef(client, tokenID)
	if err != nil {
		return err
	}
    // 创建kubeadm-cert这个secret
	err = apiclient.CreateOrUpdateSecret(client, &v1.Secret{
		ObjectMeta: metav1.ObjectMeta{
			Name:            kubeadmconstants.KubeadmCertsSecret,
			Namespace:       metav1.NamespaceSystem,
			OwnerReferences: ref,
		},
		Data: secretData,
	})
	if err != nil {
		return err
	}

	return createRBAC(client)
}

// 从磁盘上获取控制面证书，同时对控制面的证书使用key进行加密，然后保存到secretData
func getDataFromDisk(cfg *kubeadmapi.InitConfiguration, key []byte) (map[string][]byte, error) {
	secretData := map[string][]byte{}
	for certName, certPath := range certsToTransfer(cfg) {
		cert, err := loadAndEncryptCert(certPath, key)
		if err == nil || os.IsNotExist(err) {
			secretData[certOrKeyNameToSecretName(certName)] = cert
		} else {
			return nil, err
		}
	}
	return secretData, nil
}

实现机制依赖的步骤二：kubeadm master join的时候使用certificate-key解密kubeadm-cert中的内容，从而获取控制面证书

源码剖析

func runControlPlanePrepareDownloadCertsPhaseLocal(c workflow.RunData) error {
	data, ok := c.(JoinData)
	if !ok {
		return errors.New("download-certs phase invoked with an invalid data struct")
	}

	if data.Cfg().ControlPlane == nil || len(data.CertificateKey()) == 0 {
		klog.V(1).Infoln("[download-certs] Skipping certs download")
		return nil
	}

	cfg, err := data.InitCfg()
	if err != nil {
		return err
	}

	client, err := bootstrapClient(data)
	if err != nil {
		return err
	}
    // 下载kubeadm-cert secret，并用kubeadm master join的这个key来解析里面的内容
	if err := copycerts.DownloadCerts(client, cfg, data.CertificateKey()); err != nil {
		return errors.Wrap(err, "error downloading certs")
	}
	return nil
}


func DownloadCerts(client clientset.Interface, cfg *kubeadmapi.InitConfiguration, key string) error {
	fmt.Printf("[download-certs] Downloading the certificates in Secret %q in the %q Namespace\n", kubeadmconstants.KubeadmCertsSecret, metav1.NamespaceSystem)
    // 对key进行解码
	decodedKey, err := hex.DecodeString(key)
	if err != nil {
		return errors.Wrap(err, "error decoding certificate key")
	}
    // 获取kubeadm-cert这个secret
	secret, err := getSecret(client)
	if err != nil {
		return errors.Wrap(err, "error downloading the secret")
	}
    // 利用解码后的key，去解密kubeadm-cert这个secret中的数据，以获取原始的数据
	secretData, err := getDataFromSecret(secret, decodedKey)
	if err != nil {
		return errors.Wrap(err, "error decoding secret data with provided key")
	}
    // 将获取到的原始数据写入本地磁盘
	for certOrKeyName, certOrKeyPath := range certsToTransfer(cfg) {
		certOrKeyData, found := secretData[certOrKeyNameToSecretName(certOrKeyName)]
		if !found {
			return errors.Errorf("the Secret does not include the required certificate or key - name: %s, path: %s", certOrKeyName, certOrKeyPath)
		}
		if len(certOrKeyData) == 0 {
			klog.V(1).Infof("[download-certs] Not saving %q to disk, since it is empty in the %q Secret\n", certOrKeyName, kubeadmconstants.KubeadmCertsSecret)
			continue
		}
		if err := writeCertOrKey(certOrKeyPath, certOrKeyData); err != nil {
			return err
		}
	}

	return nil
}

总结

1、kubeadm init的时候会生成default bootstrap token，生成的token放在一个secret，同时指定了secret的组，以及生成对应的clusterrole和clusterrolebinding等相关的rbac对象，用于token鉴权

init的时候，bootstrap-token如果没指定，kubeadm会默认给你生成

2、join的时候：

master的kubelet直接去寻找admin的kubeconfig去用

node的kubelet会去读取cluster-info这个configmap来获取集群的ca，apiserver url等信息，
配置join token来生成bootstrap-kubelet.conf

3、看下node join的时候，这个token如何发挥作用

node join的时候，如果是control-plane的，那么bootstrap-token不会使用，直接去使用admin.conf，如果是node就需要去生成bootstrap-kubelet.conf
生成bootstrap-kubelet.conf这个kubeconfig的时候，需要的apiserver url，ca cert等从cluster-info这个configmap里面拿，然后生成的kubeconfig的auth信息里的token就是join的那个token，也就是kubeadm已经为其生成好rbac规则的那个token，后续使用这个token创建client，
就能进行node信息获取和isr的请求发起和完成认证

token的格式是tokenID.tokenSecret，因此apiserver根据我们token里tokenID找到哪个sercet，根据tokenSecret与找到的secret里面的secret字段对比，一致就认为通过。
可以使用这个secret赋予的rbac权限，这些rbac权限是在kubeadm init的时候生成的