Azure Key Vault using manage identity in AKS

Question 1

I am deploying in Azure AKS a regular deployment and i want to use keyvault to store my secrets to get access to a database.

This is my deployment file:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: sonarqube
  name: sonarqube
spec:
  selector:
    matchLabels:
      app: sonarqube
  replicas: 1
  template:
    metadata:
      labels:
        app: sonarqube
    spec:
      containers:
        - name: sonarqube
          image: sonarqube:8.9-developer
          resources:
            requests:
              cpu: 500m
              memory: 1024Mi
            limits:
              cpu: 2000m
              memory: 4096Mi
          volumeMounts:
          - mountPath: "/mnt/secrets/"
            name: secrets-store-inline
          - mountPath: "/opt/sonarqube/data/"
            name: sonar-data-new
          - mountPath: "/opt/sonarqube/extensions/plugins/"
            name: sonar-extensions-new2
          env:
          - name: "SONARQUBE_JDBC_USERNAME"
            valueFrom:
              secretKeyRef:
                name: test-secret
                key: username
          - name: "SONARQUBE_JDBC_PASSWORD"
            valueFrom:
              secretKeyRef:
                name: test-secret
                key: password
          - name: "SONARQUBE_JDBC_URL"
            valueFrom:
              configMapKeyRef:
                name: sonar-config
                key: url
          ports:
          - containerPort: 9000
            protocol: TCP
      volumes:
      - name: sonar-data-new
        persistentVolumeClaim:
          claimName: sonar-data-new
      - name: sonar-extensions-new2
        persistentVolumeClaim:
          claimName: sonar-extensions-new2
      - name: secrets-store-inline
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
           secretProviderClass: "azure-kv-provider"

and this is my secret storage class:

kind: SecretProviderClass
metadata:
  name: azure-kv-provider
spec:
  provider: azure
  secretObjects:
   - data:
      - key: username
        objectName: username
      - key: password
        objectName: password
     secretName: test-secret
     type: Opaque
  parameters:
    usePodIdentity: "false"  
    useAssignedIdentity: "true"
    userAssignedIdentityID: "zzzz-zzzz-zzzz-zzzz-zzzz"
    keyvaultName: "dbkvtz" 
    cloudName: ""   
    objects:  |
      array:
        - |
          objectName: test
          objectType: secret
          objectAlias: username
          objectVersion: "" 
        - |
          objectName: test
          objectType: secret 
          objectAlias: password  
          objectVersion: ""       
    resourceGroup: "myresourcegroup"  
    subscriptionId: "yyyy-yyyy-yyyy-yyy-yyyy"       
    tenantId: "xxxx-xxxx-xxxx-xxx-xxxx"

Where "zzzz-zzzz-zzzz-zzzz-zzzz" is the Client ID of the created Managed Identity.

In the Key Vault that i created "dbkvtz" i added through "Access Policy" the Managed Identity that i created. On the other hand in "Manage Identity" i am not able to add any role in "Azure Role Assignement" -- No role assignments found for the selected subscription. I don't know if it is necessary to add any role there.

The AKS cluster is setup for system assigned managed identity. I want to use Managed Identities to get access to the key vaults so i created a managed identity with client id "zzzz-zzzz-zzzz-zzzz-zzzz" (where is "z" a value from 0-9a-z).

I am not too familiar with keyvault integration in AKS so i am not sure if the config is ok.

I am getting this error:

kubectl describe pods:

  Normal   Scheduled    19m                  default-scheduler  Successfully assigned default/sonarqube-6bdb9cfc85-npbfw to aks-agentpool-16966606-vmss000000
  Warning  FailedMount  5m43s (x5 over 16m)  kubelet            Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[secrets-store-inline sonar-data-new sonar-extensions-new2 default-token-t45tw]: timed out waiting for the condition
  Warning  FailedMount  3m27s                kubelet            Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[default-token-t45tw secrets-store-inline sonar-data-new sonar-extensions-new2]: timed out waiting for the condition
  Warning  FailedMount  71s (x2 over 10m)    kubelet            Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[sonar-data-new sonar-extensions-new2 default-token-t45tw secrets-store-inline]: timed out waiting for the condition
  Warning  FailedMount  37s (x17 over 19m)   kubelet            MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/sonarqube-6bdb9cfc85-npbfw, err: rpc error: code = Unknown desc = failed to mount objects, error: failed to create auth config, error: failed to get credentials, nodePublishSecretRef secret is not set

logs az aks show -g RG -n SonarQubeCluster

{
  "aadProfile": null,
  "addonProfiles": {
    "azurepolicy": {
      "config": null,
      "enabled": true,
      "identity": {
        "clientId": "yy",
        "objectId": "zz",
        "resourceId": "/subscriptions/xx/resourcegroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/azurepolicy-sonarqubecluster"
      }
    },
    "httpApplicationRouting": {
      "config": null,
      "enabled": false,
      "identity": null
    },
    "omsagent": {
      "config": {
        "logAnalyticsWorkspaceResourceID": "/subscriptions/xx/resourceGroups/DefaultResourceGroup-SCUS/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-44e26024-4977-4419-8d23-0e1e22e8804e-SCUS"
      },
      "enabled": true,
      "identity": {
        "clientId": "yy",
        "objectId": "zz",
        "resourceId": "/subscriptions/xx/resourcegroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/omsagent-sonarqubecluster"
      }
    }
  },
  "agentPoolProfiles": [
    {
      "availabilityZones": [
        "1"
      ],
      "count": 2,
      "enableAutoScaling": false,
      "enableEncryptionAtHost": null,
      "enableFips": false,
      "enableNodePublicIp": null,
      "enableUltraSsd": null,
      "gpuInstanceProfile": null,
      "kubeletConfig": null,
      "kubeletDiskType": "OS",
      "linuxOsConfig": null,
      "maxCount": null,
      "maxPods": 110,
      "minCount": null,
      "mode": "System",
      "name": "agentpool",
      "nodeImageVersion": "AKSUbuntu-1804gen2containerd-2021.07.25",
      "nodeLabels": {},
      "nodePublicIpPrefixId": null,
      "nodeTaints": null,
      "orchestratorVersion": "1.20.7",
      "osDiskSizeGb": 128,
      "osDiskType": "Managed",
      "osSku": "Ubuntu",
      "osType": "Linux",
      "podSubnetId": null,
      "powerState": {
        "code": "Running"
      },
      "provisioningState": "Succeeded",
      "proximityPlacementGroupId": null,
      "scaleDownMode": null,
      "scaleSetEvictionPolicy": null,
      "scaleSetPriority": null,
      "spotMaxPrice": null,
      "tags": null,
      "type": "VirtualMachineScaleSets",
      "upgradeSettings": null,
      "vmSize": "Standard_DS2_v2"
    }
  ],
  "apiServerAccessProfile": {
    "authorizedIpRanges": null,
    "enablePrivateCluster": false,
    "enablePrivateClusterPublicFqdn": null,
    "privateDnsZone": null
  },
  "autoScalerProfile": null,
  "autoUpgradeProfile": null,
  "azurePortalFqdn": "sonarqubecluster-dns-4b5e95d4.portal.hcp.southcentralus.azmk8s.io",
  "disableLocalAccounts": null,
  "diskEncryptionSetId": null,
  "dnsPrefix": "SonarQubeCluster-dns",
  "enablePodSecurityPolicy": null,
  "enableRbac": true,
  "extendedLocation": null,
  "fqdn": "sonarqubecluster-dns-4b5e95d4.hcp.southcentralus.azmk8s.io",
  "fqdnSubdomain": null,
  "httpProxyConfig": null,
  "id": "/subscriptions/xx/resourcegroups/RG/providers/Microsoft.ContainerService/managedClusters/SonarQubeCluster",
  "identity": {
    "principalId": "yy",
    "tenantId": "rr",
    "type": "SystemAssigned",
    "userAssignedIdentities": null
  },
  "identityProfile": {
    "kubeletidentity": {
      "clientId": "yy",
      "objectId": "zz",
      "resourceId": "/subscriptions/xx/resourcegroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/SonarQubeCluster-agentpool"
    }
  },
  "kubernetesVersion": "1.20.7",
  "linuxProfile": null,
  "location": "southcentralus",
  "maxAgentPools": 100,
  "name": "SonarQubeCluster",
  "networkProfile": {
    "dnsServiceIp": "10.0.0.10",
    "dockerBridgeCidr": "172.17.0.1/16",
    "loadBalancerProfile": {
      "allocatedOutboundPorts": null,
      "effectiveOutboundIPs": [
        {
          "id": "/subscriptions/xx/resourceGroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.Network/publicIPAddresses/nn",
          "resourceGroup": "MC_xx_SonarQubeCluster_southcentralus"
        }
      ],
      "idleTimeoutInMinutes": null,
      "managedOutboundIPs": {
        "count": 1
      },
      "outboundIPs": null,
      "outboundIpPrefixes": null
    },
    "loadBalancerSku": "Standard",
    "natGatewayProfile": null,
    "networkMode": null,
    "networkPlugin": "kubenet",
    "networkPolicy": null,
    "outboundType": "loadBalancer",
    "podCidr": "10.244.0.0/16",
    "serviceCidr": "10.0.0.0/16"
  },
  "nodeResourceGroup": "MC_xx_SonarQubeCluster_southcentralus",
  "podIdentityProfile": null,
  "powerState": {
    "code": "Running"
  },
  "privateFqdn": null,
  "privateLinkResources": null,
  "provisioningState": "Succeeded",
  "resourceGroup": "RG",
  "securityProfile": null,
  "servicePrincipalProfile": {
    "clientId": "msi"
  },
  "sku": {
    "name": "Basic",
    "tier": "Free"
  },
  "type": "Microsoft.ContainerService/ManagedClusters",
  "windowsProfile": null
}

Any idea of what is wrong?

Thank you in advance.

Question 2

The userAssignedIdentityID in your SecretProviderClass must be the User-assigned Kubelet managed identity ID (Managed Identity for the NodePool) and not the Managed Identity created for your AKS bcs the volumes will be access via kubelet on the nodes.

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: azure-kvname-user-msi
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true"
    userAssignedIdentityID: "<Kubelet identity ID>"
    keyvaultName: "kvname"

You also need to assign a Role to this Kubelet Identity:

resource "azurerm_role_assignment" "akv_kubelet" {
  scope                = azurerm_key_vault.akv.id
  role_definition_name = "Key Vault Secrets Officer"
  principal_id         = azurerm_kubernetes_cluster.aks.kubelet_identity[0].object_id
}

or

export KUBE_ID=$(az aks show -g <resource group> -n <aks cluster name> --query identityProfile.kubeletidentity.objectId -o tsv)
export AKV_ID=$(az keyvault show -g <resource group> -n <akv name> --query id -o tsv)
az role assignment create --assignee $KUBE_ID --role "Key Vault Secrets Officer" --scope $AKV_ID

Documentation can be found here for user-assigned identity and here for system-assigned identity.

Azure Key Vault using manage identity in AKS

K8SOQ

Answer a question

Answers

所有评论(0)

K8SOQ