---

权限认证

一、kubernetes权限RBAC

kubernetes使用的RBAC的示意图如下：user1 可以使用role1和role2所允许的资源，user2只可以使用 role1 所允许的资源。我们有时候会称user为subject

在kubernetes，serviceaccount、user、group为user。、clusterrole和role代表role。clusterrolebinding和rolebinding代表rolebinding。

• GlobalRole（CRD资源）和ClusterRole、Role：全局角色，集群角色，项目角色，这三个都包含Rules，定义了这个Role所能允许的资源。三者的区别在于，在Role中是有namespace的，属于单个命名空间，ClusterRole是单个集群的，GlobalRole是多个集群的。
• Subject ：里面对应相应的用户如上图所示，Subject里面包含：apiGroup、Kind、Name（User的Name）
• RoleBinding、ClusterRoleBinding、GlobalRoleBinding：将Role与Subject绑定。确定每一个用户都有确定的Rules（权限）。

二、验证权限的过程

1. 填充请求。确定请求的范围，单集群还是多集群，是否是发给k8s的请求
2. 验证用户，密码、token等方式，如果是无用户则定义为anonymous
3. 如果是多集群，则需要转发。
4. 确定该资源是否能被用户获取
   1. 通过user，globalRoleBinding获取到globalRole，查看GlobalRole对该资源是否允许。
   2. 通过user，ClusterroleBinding获取到Clusterrole
   3. 通过user，Rolebinding获取到Role，查看Role对资源是否允许
   为了减少查询次数，通过if来过滤
5. 审计
6. 如果是k8s资源，类似于kubectl get pod -A的资源，则转发给kube-apiserver

三、代码分析

本项目使用的是iam模块来进行权限和权限获取，本项目使用go-restful开发 web应用框架，权限认证和权限获取都是在filter进行的
权限代码如下： **注意：filter和handle的方式一样，先被封装后被运行。**所以下面请求的执行过程是：
WithRequestInfo—>WithAuthentication—>WithMultipleClusterDispatcher—>WithAuthorization—>WithAuditing---->WithKubeAPIServer

func (s *APIServer) buildHandlerChain(stopCh <-chan struct{}) {
	//.....
	handler := s.Server.Handler
	handler = filters.WithKubeAPIServer(handler, s.KubernetesClient.Config(), &errorResponder{})
//开启审计
	if s.Config.AuditingOptions.Enable {
		handler = filters.WithAuditing(handler,
			audit.NewAuditing(s.InformerFactory, s.Config.AuditingOptions.WebhookUrl, stopCh))
	}
	//授权
	handler = filters.WithAuthorization(handler, authorizers)
	if s.Config.MultiClusterOptions.Enable {
		//多集群转发
		handler = filters.WithMultipleClusterDispatcher(handler, clusterDispatcher)
	}

	loginRecorder := im.NewLoginRecorder(s.KubernetesClient.KubeSphere())
	//认证
	handler = filters.WithAuthentication(handler, authn, loginRecorder)
	//WithRequestInfo
	handler = filters.WithRequestInfo(handler, requestInfoResolver)
	s.Server.Handler = handler
}

主要分为以下步骤

1. WithRequestInfo ：填充请求信息
2. WithAuthentication：权限认证
3. WithMultipleClusterDispatcher：多集群路径填补和转发，多集群时根据requestinfo，将信息发送给对应的集群。
4. WithAuthorization：权限获取，判断是否对应集群资源有获取权限。
5. WithAuditing：审计
6. WithKubeAPIServer：转发，如果是直接对k8s的请求，则直接转发给k8s。

四、填充请求信息

请求信息如下：

type RequestInfo struct {
    // import k8s.io/apiserver/pkg/endpoints/request
	*k8srequest.RequestInfo
	//是否是k8s的请求，如果是k8s的请求，则会直接转发到kube-apiserver
	IsKubernetesRequest bool

	//请求资源所在的Workspace，可以为空
	Workspace string

	//请求资源所在的Cluster，如果是单集群则为空
	Cluster string

	//请求资源所在的devops项目
	DevOps string

	// 资源请求范围
	ResourceScope string
}

认证
有三个认证方式，anoymous、password、token三个方法任意一个通过校验，就立刻返回。

1. anoymous的实现方法如下

 func (a *Authenticator) AuthenticateRequest(req *http.Request) (*authenticator.Response, bool, error) {
   	auth := strings.TrimSpace(req.Header.Get("Authorization"))
   	if auth == "" {
   		return &authenticator.Response{
   			User: &user.DefaultInfo{
   				Name:   user.Anonymous,
   				UID:    "",
   				Groups: []string{user.AllUnauthenticated},
   			},
   		}, true, nil
   	}
   	return nil, false, nil
   }

2. password的实现方法如下：从user资源中获取信息

func (im *passwordAuthenticator) Authenticate(username, password string) (authuser.Info, error) {
   	//从k8s 集群中查询到user资源
   	user, err := im.searchUser(username)
   	
   	providerOptions, ldapProvider := im.getLdapProvider()
   	//....
       //通过ldapProvider来判断账号密码是否正确
   	if ldapProvider != nil && username != constants.AdminUserName {
   		//
   		authenticated, err := ldapProvider.Authenticate(username, password)
   		//....
   		if authenticated != nil {
   			return &authuser.DefaultInfo{
   				Name: authenticated.Name,
   				UID:  string(authenticated.UID),
   			}, nil
   		}
   	}
   	//也可以通过use资源的EncryptedPassword，验证密码
   	if checkPasswordHash(password, user.Spec.EncryptedPassword) {
   		return &authuser.DefaultInfo{
   			Name: user.Name,
   			UID:  string(user.UID),
   		}, nil
   	}
   
   	return nil, AuthFailedIncorrectPassword
   }

3. token：使用jwt验证

  func (t tokenOperator) Verify(tokenStr string) (user.Info, error) {
       //调用jwt包验证
   	authenticated, tokenType, err := t.issuer.Verify(tokenStr)
   	//这里应该是过期时间，0为永不过期。这个为配置设置
   	if t.options.OAuthOptions.AccessTokenMaxAge == 0 ||
   		tokenType == token.StaticToken {
   		return authenticated, nil
   	}
       //redis验证
   	if err := t.tokenCacheValidate(authenticated.GetName(), tokenStr); err != nil {
   		klog.Error(err)
   		return nil, err
   	}
   	return authenticated, nil
   }

五、多集群路由转发和协议升级

只在多集群下运行此函数

func (c *clusterDispatch) Dispatch(w http.ResponseWriter, req *http.Request, handler http.Handler) {
	info, _ := request.RequestInfoFrom(req.Context())
	//cluster 是crd 资源。获取集群所有cluster信息
	cluster, err := c.clusterLister.Get(info.Cluster)
	//请求集群是主机集群，不需要通过代理
	if isClusterHostCluster(cluster) {
		req.URL.Path = strings.Replace(req.URL.Path, fmt.Sprintf("/clusters/%s", info.Cluster), "", 1)
		handler.ServeHTTP(w, req)
		return
	}
	//查询集群
	innCluster := c.getInnerCluster(cluster.Name)
	transport := http.DefaultTransport
	//替换url
	u := *req.URL
	u.Path = strings.Replace(u.Path, fmt.Sprintf("/clusters/%s", info.Cluster), "", 1)
	if cluster.Spec.Connection.Type == clusterv1alpha1.ConnectionTypeDirect &&
		len(cluster.Spec.Connection.KubeSphereAPIEndpoint) == 0 {
		u.Scheme = innCluster.kubernetesURL.Scheme
		u.Host = innCluster.kubernetesURL.Host
		u.Path = fmt.Sprintf(proxyURLFormat, u.Path)
		transport = innCluster.transport
         //...
	} else {
		u.Host = innCluster.kubesphereURL.Host
		u.Scheme = innCluster.kubesphereURL.Scheme
	}
    
	httpProxy := proxy.NewUpgradeAwareHandler(&u, transport, false, false, c)
	httpProxy.ServeHTTP(w, req)
}

六、权限获取

首先在apiserver层中初始化Authorizer，然后进行授权模式的判断 代码如下：

var authorizers authorizer.Authorizer

	switch a.Config.AuthorizationOptions.Mode{
	case authorizationoptions.AlwaysAllow:
		authorizers = authorizerfactory.NewAlwaysAllowAuthorizer()
	case authorizationoptions.AlwaysDeny:
		authorizers = authorizerfactory.NewAlwaysDenyAuthorizer()
	default:
		fallthrough
	case authorizationoptions.RBAC:
	//放行的所有路径：如登录等
		excludePaths := []string{"/login/*"}
		pathAuthorizer,_ :=path.NewAuthorizer(excludePaths)
//RBAC权限判断，url鉴权
		RBACAuthorizer:=rbac.NewRBACAuthorizer(a.KubernetesClient.Kubernetes(),a.KubernetesClient.Kubesphere())
		//将放行的路径和RBAC权限授权放到unionauthorizer里面，然后进行首先判断，是不是放行的路径，根据pathAuthorizer返回的Authorizer，来判断如果是放行的路径，直接放行，否则进行RBAC权限判定
		authorizers = unionauthorizer.New(pathAuthorizer,RBACAuthorizer)
	}
//将授权加到handler里面
	handler = middleware.WithAuthorization(handler,authorizers)

path放行路径的截取
代码如下：

	var prefixes []string
	paths := sets.NewString()
	for _, p := range alwaysAllowPaths {
	//将放行的URL的前缀/去掉，如：/login/*去掉后为login/*
		p = strings.TrimPrefix(p, "/")
		if len(p) == 0 {
			paths.Insert(p)
			continue
		}
		//判断放行的URL是不是写的为/**
		if strings.ContainsRune(p[:len(p)-1], '*') {
			return nil, fmt.Errorf("only trailing * allowed in %q", p)
		}
		//去掉后面的*
		if strings.HasSuffix(p, "*") {
			prefixes = append(prefixes, p[:len(p)-1])
		} else {
			paths.Insert(p)
		}
	}
	return authorizer.AuthorizerFunc(func(a authorizer.Attributes) (authorizer.Decision, string, error) {
		pth := strings.TrimPrefix(a.GetPath(), "/")
		//如果是不带*的则需要用户请求的路径和放行的路径完全相同
		if paths.Has(pth) {
			return authorizer.DecisionAllow, "", nil
		}
		//如果是/*的则只要前缀为例如/login的即可
		for _, prefix := range prefixes {
			if strings.HasPrefix(pth, prefix) {
			//放行
				return authorizer.DecisionAllow, "", nil
			}
		}

		return authorizer.DecisionNoOpinion, "", nil
	}), nil

Union来判断是放行的路径，还是进行RBAC授权

//...authorizer.Authorizer 代表可以存放多个类型为Authorizer的结构体（即：放行的路径、RBAC授权）
func New(authorizationHandlers ...authorizer.Authorizer) authorizer.Authorizer {
	return unionAuthzHandler(authorizationHandlers)
}

func (authzHandler unionAuthzHandler) Authorize(a authorizer.Attributes) (authorizer.Decision, string, error) {
	var (
		errlist    []error
		reasonlist []string
	)

	for _, currAuthzHandler := range authzHandler {
		decision, reason, err := currAuthzHandler.Authorize(a)

		if err != nil {
			errlist = append(errlist, err)
		}
		if len(reason) != 0 {
			reasonlist = append(reasonlist, reason)
		}
		//根据截取放行的路径返回的放行DecisionAllow，进行放行
		switch decision {
		case authorizer.DecisionAllow, authorizer.DecisionDeny:
			return decision, reason, err
		case authorizer.DecisionNoOpinion:
			// continue to the next authorizer
		}
	}

	return authorizer.DecisionNoOpinion, strings.Join(reasonlist, "\n"), utilerrors.NewAggregate(errlist)
}

RBAC用户授权

func appliesTo(user user.Info, bindingSubjects []rbacv1.Subject, namespace string) (int, bool) {
	for i, bindingSubject := range bindingSubjects {
		if appliesToUser(user, bindingSubject, namespace) {
			return i, true
		}
	}
	return 0, false
}
//根据请求的用户去查看是否有该用户
func appliesToUser(user user.Info, subject rbacv1.Subject, namespace string) bool {
	switch subject.Kind {
	case rbacv1.UserKind:
		a :=strings.Compare(user.GetName(),subject.Name)
		return a == 0
	default:
		return false
	}
}

RBAC权限匹配

func ruleAllows(requestAttributes authorizer.Attributes, rule *rbacv1.PolicyRule) bool {
	//判断是否是资源请求
	if requestAttributes.IsResourceRequest() {
		//如果含有子资源的请求，则进行拼接
		combinedResource := requestAttributes.GetResource()
		if len(requestAttributes.GetSubresource()) > 0 {
			combinedResource = requestAttributes.GetResource() + "/" + requestAttributes.GetSubresource()
		}
		//进行权限、APIGroup、资源、资源名字进行判断用户请求的URL和该用户拥有的权限是否一致
		return VerbMatches(rule, requestAttributes.GetVerb()) &&
			APIGroupMatches(rule, requestAttributes.GetAPIGroup()) &&
			ResourceMatches(rule, combinedResource, requestAttributes.GetSubresource()) &&
			ResourceNameMatches(rule, requestAttributes.GetName())
	}
	return VerbMatches(rule, requestAttributes.GetVerb())
}

权限获取是通过RBAC方式。先调用Authorize

func (r *RBACAuthorizer) Authorize(requestAttributes authorizer.Attributes) (authorizer.Decision, string, error) {
	//调用visitRulesFor检查权限
	r.visitRulesFor(requestAttributes, ruleCheckingVisitor.visit)
	if ruleCheckingVisitor.allowed {
		return authorizer.DecisionAllow, ruleCheckingVisitor.reason, nil
	}
    //...
    //记录日志
    klog.Infof("...")
    //...
	return authorizer.DecisionNoOpinion, reason, nil
}

在调用vistRulesFor

func (r *RBACAuthorizer) visitRulesFor(requestAttributes authorizer.Attributes, visitor func(source fmt.Stringer, regoPolicy string, rule *rbacv1.PolicyRule, err error) bool) {
	//查看globalRole是否有对此资源的获取权限
	if globalRoleBindings, err := r.am.ListGlobalRoleBindings(""); err != nil {
		//...
	} else {
		sourceDescriber := &globalRoleBindingDescriber{}
        //循环globalRoleBindings。找到该用户的globalRoleBindings。
		for _, globalRoleBinding := range globalRoleBindings {
			subjectIndex, applies := appliesTo(requestAttributes.GetUser(), globalRoleBinding.Subjects, "")
			if !applies {
                //如果不是这个找到该用户的globalRoleBindings，则continue
				continue
			}
            //根据globalRoleBinding上的roleref，获取到role。进一步获取到regoPolicy和rules。这是两个规则，只要有一个规则符合即可
			regoPolicy, rules, err := r.am.GetRoleReferenceRules(globalRoleBinding.RoleRef, "")
			//...
             //根据regoPolicy验证是否符合规则
			if !visitor(sourceDescriber, regoPolicy, nil, nil) {
				return
			}
             //根据rules验证是否符合规则
			for i := range rules {
				if !visitor(sourceDescriber, "", &rules[i], nil) {
					return
				}
			}
		}
        //...
	}
	
	if requestAttributes.GetResourceScope() == request.WorkspaceScope ||
		requestAttributes.GetResourceScope() == request.NamespaceScope ||
		requestAttributes.GetResourceScope() == request.DevOpsScope {
		//...
         // 代码部分删除，这部分主要是获取workspace
	    workspace, err = r.am.GetNamespaceControlledWorkspace(requestAttributes.GetNamespace()); err != nil 
		// 通过workspace获取workspaceRoleBindings。
		if workspaceRoleBindings, err := r.am.ListWorkspaceRoleBindings("", workspace); ...{
		     //...
		} else {
			//...轮训workspaceRoleBindings，找到workspaceRole，在进行权限检测。这部分和grobalrole一样
			for _, workspaceRoleBinding := range workspaceRoleBindings {
				subjectIndex, applies := appliesTo(requestAttributes.GetUser(), workspaceRoleBinding.Subjects, "")
				if !applies {
					continue
				}
				regoPolicy, rules, err := r.am.GetRoleReferenceRules(workspaceRoleBinding.RoleRef, "")
				//...
				if !visitor(sourceDescriber, regoPolicy, nil, nil) {
					return
				}
				for i := range rules {
					if !visitor(sourceDescriber, "", &rules[i], nil) {
						return
					}
				}
			}
		}
	}

	if requestAttributes.GetResourceScope() == request.NamespaceScope ||
		requestAttributes.GetResourceScope() == request.DevOpsScope {
		
		namespace := requestAttributes.GetNamespace()
		// 直接获取namespace，或者根据DevOps获取namespace
		if requestAttributes.GetResourceScope() == request.DevOpsScope {
			if relatedNamespace, err := r.am.GetDevOpsRelatedNamespace(requestAttributes.GetDevOps()); err != nil {
				if !visitor(nil, "", nil, err) {
					return
				}
			} else {
				namespace = relatedNamespace
			}
		}
		//根据namespace获取rolebinding
		if roleBindings, err := r.am.ListRoleBindings("", namespace); err != nil {
			if !visitor(nil, "", nil, err) {
				return
			}
		} else {
			sourceDescriber := &roleBindingDescriber{}
             //轮训roleBindings，找到role,检查role
			for _, roleBinding := range roleBindings {
				subjectIndex, applies := appliesTo(requestAttributes.GetUser(), roleBinding.Subjects, namespace)
				if !applies {
					continue
				}
				regoPolicy, rules, err := r.am.GetRoleReferenceRules(roleBinding.RoleRef, namespace)
				if err != nil {
					visitor(nil, "", nil, err)
					continue
				}
				sourceDescriber.binding = roleBinding
				sourceDescriber.subject = &roleBinding.Subjects[subjectIndex]
				if !visitor(sourceDescriber, regoPolicy, nil, nil) {
					return
				}
				for i := range rules {
					if !visitor(sourceDescriber, "", &rules[i], nil) {
						return
					}
				}
			}
		}
	}
	//获取所有clusterRoleBindings，轮训查找到clusterRole。检查规则
	if clusterRoleBindings, err := r.am.ListClusterRoleBindings(""); err != nil {
		if !visitor(nil, "", nil, err) {
			return
		}
	} else {
		sourceDescriber := &clusterRoleBindingDescriber{}
		for _, clusterRoleBinding := range clusterRoleBindings {
			subjectIndex, applies := appliesTo(requestAttributes.GetUser(), clusterRoleBinding.Subjects, "")
			if !applies {
				continue
			}
			regoPolicy, rules, err := r.am.GetRoleReferenceRules(clusterRoleBinding.RoleRef, "")
			if err != nil {
				visitor(nil, "", nil, err)
				continue
			}
			sourceDescriber.binding = clusterRoleBinding
			sourceDescriber.subject = &clusterRoleBinding.Subjects[subjectIndex]
			if !visitor(sourceDescriber, regoPolicy, nil, nil) {
				return
			}
			for i := range rules {
				if !visitor(sourceDescriber, "", &rules[i], nil) {
					return
				}
			}
		}
	}
}

检查规则的函数是visitor，如果regoPolicy符合就不检查rule

func (v *authorizingVisitor) visit(source fmt.Stringer, regoPolicy string, rule *rbacv1.PolicyRule, err error) bool {
    //调用open-policy-agent库检查权限
	if regoPolicy != "" && regoPolicyAllows(v.requestAttributes, regoPolicy) {
		v.allowed = true
		v.reason = fmt.Sprintf("RBAC: allowed by %s", source.String())
		return false
	}
    //调用k8s 接口实现rbac检查权限
	if rule != nil && ruleAllows(v.requestAttributes, rule) {
		v.allowed = true
		v.reason = fmt.Sprintf("RBAC: allowed by %s", source.String())
		return false
	}
	if err != nil {
		v.errors = append(v.errors, err)
	}
	return true
}

七、k8s集群资源转发

如果这个资源是向k8s请求的，则直接向k8s请求

func WithKubeAPIServer(handler http.Handler, config *rest.Config, failed proxy.ErrorResponder) http.Handler {
	//...
    //这部分为初始化内容，在请求时不执行。
	return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
		info, ok := request.RequestInfoFrom(req.Context())
		if !ok {
			err := errors.New("Unable to retrieve request info from request")
			klog.Error(err)
			responsewriters.InternalError(w, req, err)
		}
		// 确认是否为k8s请求
		if info.IsKubernetesRequest {
			s := *req.URL
			s.Host = kubernetes.Host
			s.Scheme = kubernetes.Scheme

			// make sure we don't override kubernetes's authorization
			req.Header.Del("Authorization")
            //转发
			httpProxy := proxy.NewUpgradeAwareHandler(&s, defaultTransport, true, false, failed)
			httpProxy.UpgradeTransport = proxy.NewUpgradeRequestRoundTripper(defaultTransport, defaultTransport)
			httpProxy.ServeHTTP(w, req)
			return
		}

		handler.ServeHTTP(w, req)
	})
}

总结

希望与大佬们进行交流学习 。如有侵权部分请联系删除侵权内容