I am trying to explore vault enterprise but getting permission denied for sidecar when I use the vault enterprise but seems to work fine when I tried to use local vault server.

Here is the repository that contains a working example with the local vault vault-sidecar-injector-app

Vault config

export VAULT_ADDR="https://vault-cluster.vault.c1c633fa-91ef-4e86-b025-4f31b3f14730.aws.hashicorp.cloud:8200"
export VAULT_NAMESPACE="admin"

#install agent
helm upgrade --install vault hashicorp/vault --set "injector.externalVaultAddr=$VAULT_ADDR"
vault auth enable kubernetes
# get certs & host
VAULT_HELM_SECRET_NAME=$(kubectl get secrets --output=json | jq -r '.items[].metadata | select(.name|startswith("vault-token-")).name')
TOKEN_REVIEW_JWT=$(kubectl get secret $VAULT_HELM_SECRET_NAME --output='go-template={{ .data.token }}' | base64 --decode)
KUBE_CA_CERT=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode)
KUBE_HOST=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')

# set Kubernetes config
vault write auth/kubernetes/config \
        token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
        kubernetes_host="$KUBE_HOST" \
        kubernetes_ca_cert="$KUBE_CA_CERT" \
        issuer="https://kubernetes.default.svc.cluster.local" \
        disable_iss_validation="true" \
        disable_local_ca_jwt="true"


vault auth enable approle
# create admin policy
vault policy write admin admin-policy.hcl
vault write auth/approle/role/admin policies="admin"
vault read auth/approle/role/admin/role-id
# generate secret
vault write -f  auth/approle/role/admin/secret-id
#Enable KV
vault secrets enable -version=2 kv

I can see the role and policy

Admin policy

Here is the admin policy for the enterprise

path "*" {
capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

Deploy Script for helm

here is the deploy script, tried hcp-root root policy but no luck

RELEASE_NAME=demo-managed
NAMESPACE=default
ENVIRONMENT=develop
export role_id="f9782a53-823e-2c08-81ae-abc"
export secret_id="1de3b8c5-18c7-60e3-24ca-abc"
export VAULT_ADDR="https://vault-cluster.vault.c1c633fa-91ef-4e86-b025-4f31b3f14730.aws.hashicorp.cloud:8200"
export VAULT_TOKEN=$(vault write -field="token"  auth/approle/login role_id="${role_id}" secret_id="${secret_id}")
vault write auth/kubernetes/role/${NAMESPACE}-${RELEASE_NAME} bound_service_account_names=${RELEASE_NAME} bound_service_account_namespaces=${NAMESPACE} policies=hcp-root ttl=1h
helm upgrade --install $RELEASE_NAME ../helm-chart --set environment=$ENVIRONMENT --set nameOverride=$RELEASE_NAME

also tried with root token

RELEASE_NAME=demo-managed
NAMESPACE=default
ENVIRONMENT=develop
vault write auth/kubernetes/role/${NAMESPACE}-${RELEASE_NAME} bound_service_account_names=${RELEASE_NAME} bound_service_account_namespaces=${NAMESPACE} policies=hcp-root ttl=1h
helm upgrade --install $RELEASE_NAME ../helm-chart --set environment=$ENVIRONMENT --set nameOverride=$RELEASE_NAME

Sidecar config

With namespace annotation, as my understanding namespace is required

vault.hashicorp.com/namespace - configures the Vault Enterprise namespace to be used when requesting secrets from Vault.

https://www.vaultproject.io/docs/platform/k8s/injector/annotations

vault.hashicorp.com/namespace : "admin"

Error

  | Error making API request.
  |
  | URL: PUT https://vault-cluster.vault.c1c633fa-91ef-4e86-b025-4f31b3f14730.aws.hashicorp.cloud:8200/v1/admin/auth/kubernetes/login
  | Code: 403. Errors:
  |
  | * permission denied

Without namespace annotation getting below error

  | URL: PUT https://vault-cluster.vault.c1c633fa-91ef-4e86-b025-4f31b3f14730.aws.hashicorp.cloud:8200/v1/auth/kubernetes/login
  | Code: 400. Errors:
  |
  | * missing client token

Even enabling debug logs vault.hashicorp.com/log-level : "debug" does not help me with this error, any help or suggestions will be appreciated.

Also tried https://support.hashicorp.com/hc/en-us/articles/4404389946387-Kubernetes-auth-method-Permission-Denied-error

So seems like I am missing something very specific to the vault enterprise

Finally able to resolve the weird issue with vault, posting as an answer might help someone else.

The only thing that I missed to understand the flow between vault server, sidecar, and Kubernetes.

Kubernetes should be reachable to vault enterprise for Token review API calls. As you can see the when sidecar makes a request to the vault, then the vault enterprise server performs a token review API call.

Use the /config endpoint to configure Vault to talk to Kubernetes. Use kubectl cluster-info to validate the Kubernetes host address and TCP port.

https://www.vaultproject.io/docs/auth/kubernetes

  | Error making API request.
  |
  | URL: PUT https://vault-cluster.vault.c1c633fa-91ef-4e86-b025-4f31b3f14730.aws.hashicorp.cloud:8200/v1/admin/auth/kubernetes/login
  | Code: 403. Errors:
  |
  | * permission denied
   backoff=2.99s

This error does not indicate it has a connectivity issue but this also happens when the vault is not able to communicate with the Kubernetes cluster.

Kube Host

vault write auth/kubernetes/config \
        token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
        kubernetes_host="$KUBE_HOST" \
        kubernetes_ca_cert="$KUBE_CA_CERT" \
        issuer="https://kubernetes.default.svc.cluster.local"

KUBE_HOST should be reachable for vault enterprise for tokenreview process.

So for the vault to communicate with our cluster, we need a few changes.

minikube start --apiserver-ips=14.55.145.30 --vm-driver=none

Now update the vaul-config.sh file

KUBE_HOST=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')

change this to

KUBE_HOST=""https://14.55.145.30:8443/"

No manual steps, for the first time configuration run

./vault-config.sh

and for the rest of the deployment in your CI/CD you can use

./vault.sh

Each release has only been able to access its own secrets.

Furter details can be found start-minikube-in-ec2

TLDR,

Note: Kubernetes cluster should be reachable to vault enterprise for authentication, so vault enterprise would not able to communicate with your local minikube cluster. Better to test it out on EC2