Answer a question

I'm using AWS EKS 1.21 with service account discovery enabled.
Created an OIDC provider, the .well-known/openid-configuration endpoint returns a correct configuration:

{
  "issuer": "https://oidc.eks.eu-west-1.amazonaws.com/id/***",
  "jwks_uri": "https://ip-***.eu-west-1.compute.internal:443/openid/v1/jwks",
  "response_types_supported": [
    "id_token"
  ],
  "subject_types_supported": [
    "public"
  ],
  "id_token_signing_alg_values_supported": [
    "RS256"
  ]
}

Created a ServiceAccount for one of my deployments and the pod gets this as projected volume:

  volumes:
  - name: kube-api-access-b4xt9
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace

The secret created for the ServiceAccount contains this token:

{
  "iss": "kubernetes/serviceaccount",
  "kubernetes.io/serviceaccount/namespace": "sbx",
  "kubernetes.io/serviceaccount/secret.name": "dliver-site-config-service-token-kz874",
  "kubernetes.io/serviceaccount/service-account.name": "dliver-site-config-service",
  "kubernetes.io/serviceaccount/service-account.uid": "c26ad760-9067-4d90-a327-b3d6e32bce42",
  "sub": "system:serviceaccount:sbx:dliver-site-config-service"
}

The projected token mounted in to the pod contains this:

{
  "aud": [
    "https://kubernetes.default.svc"
  ],
  "exp": 1664448004,
  "iat": 1632912004,
  "iss": "https://oidc.eks.eu-west-1.amazonaws.com/id/***",
  "kubernetes.io": {
    "namespace": "sbx",
    "pod": {
      "name": "dliver-site-config-service-77494b8fdd-45pxw",
      "uid": "0dd440a6-1213-4faa-a69e-398b83d2dd6b"
    },
    "serviceaccount": {
      "name": "dliver-site-config-service",
      "uid": "c26ad760-9067-4d90-a327-b3d6e32bce42"
    },
    "warnafter": 1632915611
  },
  "nbf": 1632912004,
  "sub": "system:serviceaccount:sbx:dliver-site-config-service"
}

Kubernetes renew the projected token every hour, so everything looks fine.
Except the projected token "exp" field:
"iat": 1632912004 which is Wednesday, September 29, 2021 10:40:04 AM
"exp": 1664448004 which is Thursday, September 29, 2022 10:40:04 AM

So the problem is, that the projected token expiry time is 1 year, instead of around 1 hour, which makes Kubernetes effort to renew the token basically useless.
I searched for hours but was simply unable to figure out where this is coming from.
The expiration flag is passed to the kube-api server: --service-account-max-token-expiration="24h0m0s", so my assumption is that this should be configured on the OIDC provider somehow, but unable to find any related documentation.

Any idea how to make the projected token expiry date around the same as the expirationSeconds in the pod projected volume?

Update

It only happens, when the projected token expirationSeconds is set to the default 3607 value, any other value gives the correct exp in the mounted token, which is really weird.

Answers

Finally got an answer elsewhere.

cluster operators can specify flag --service-account-extend-token-expiration=true to kube apiserver to allow tokens have longer expiration temporarily during the migration. Any usage of legacy token will be recorded in both metrics and audit logs.

The "3607" magic number is part of the Bound Service Account Tokens safe rollout plan, described in this kep. The actual number hardcoded in the source code.
The --service-account-extend-token-expiration flag was set to true by default from 1.20.

The mentioned metric/log info can be found in the kep too and was implemented here.
To see these logs in EKS need to enable audit logging on the cluster, then check Cloudwatch for related log entries.

I used this query in Cloudwatch Log Insight to find which pods don't reload the token periodically:

filter @logStream like 'kube-apiserver-audit'
 | filter ispresent(`annotations.authentication.k8s.io/stale-token`)
 | parse `annotations.authentication.k8s.io/stale-token` "subject: *," as subject
 | stats count(*) as staleCount by subject, `user.username`
 | sort staleCount desc
Logo

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐