I have a baremetal cluster deployed using Kubespray with kubernetes 1.22.2, MetalLB, and ingress-nginx enabled. I am getting 404 Not found when trying to access any service deployed via helm when setting ingressClassName: nginx. However, everything works fine if I don't use ingressClassName: nginx but kubernetes.io/ingress.class: nginx instead in the helm chart values.yaml. How can I get it to work using ingressClassName?

These are my kubespray settings for inventory/mycluster/group_vars/k8s_cluster/addons.yml

# Nginx ingress controller deployment
ingress_nginx_enabled: true
ingress_nginx_host_network: false
ingress_publish_status_address: ""
ingress_nginx_nodeselector:
  kubernetes.io/os: "linux"
ingress_nginx_tolerations:
  - key: "node-role.kubernetes.io/master"
    operator: "Equal"
    value: ""
    effect: "NoSchedule"
  - key: "node-role.kubernetes.io/control-plane"
    operator: "Equal"
    value: ""
    effect: "NoSchedule"
ingress_nginx_namespace: "ingress-nginx"
ingress_nginx_insecure_port: 80
ingress_nginx_secure_port: 443
ingress_nginx_configmap:
  map-hash-bucket-size: "128"
  ssl-protocols: "TLSv1.2 TLSv1.3"
ingress_nginx_configmap_tcp_services:
  9000: "default/example-go:8080"
ingress_nginx_configmap_udp_services:
  53: "kube-system/coredns:53"
ingress_nginx_extra_args:
  - --default-ssl-certificate=default/mywildcard-tls
ingress_nginx_class: "nginx"

grafana helm values.yaml

ingress:
  enabled: true
  # For Kubernetes >= 1.18 you should specify the ingress-controller via the field ingressClassName
  # See https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#specifying-the-class-of-an-ingress
  ingressClassName: nginx
  # Values can be templated
  annotations:
  #  kubernetes.io/ingress.class: nginx
  #  kubernetes.io/tls-acme: "true"
  labels: {}
  path: /

  # pathType is only for k8s >= 1.1=
  pathType: Prefix

  hosts:
    - grafana.mycluster.org
  tls:
   - secretName: mywildcard-tls
     hosts:
       - grafana.mycluster.org

kubectl describe pod grafana-679bbfd94-p2dd7

...
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  25m                default-scheduler  Successfully assigned default/grafana-679bbfd94-p2dd7 to node1
  Normal   Pulled     25m                kubelet            Container image "grafana/grafana:8.2.2" already present on machine
  Normal   Created    25m                kubelet            Created container grafana
  Normal   Started    25m                kubelet            Started container grafana
  Warning  Unhealthy  24m (x3 over 25m)  kubelet            Readiness probe failed: Get "http://10.233.90.33:3000/api/health": dial tcp 10.233.90.33:3000: connect: connection refused

kubectl get svc

NAME         TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
grafana      LoadBalancer   10.233.14.90   10.10.30.52   80:30285/TCP   55m
kubernetes   ClusterIP      10.233.0.1     <none>        443/TCP        9d

kubectl get ing (no node address assigned)

NAME      CLASS   HOSTS                    ADDRESS   PORTS     AGE
grafana   nginx   grafana.mycluster.org             80, 443   25m

kubectl describe ing grafana (no node address assigned)

Name:             grafana
Namespace:        default
Address:
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
  mywildcard-tls terminates grafana.mycluster.org
Rules:
  Host                    Path  Backends
  ----                    ----  --------
  grafana.mycluster.org
                          /   grafana:80 (10.233.90.33:3000)
Annotations:              meta.helm.sh/release-name: grafana
                          meta.helm.sh/release-namespace: default
Events:                   <none>

kubectl get all --all-namespaces

NAMESPACE        NAME                                                              READY   STATUS    RESTARTS   AGE
default          pod/grafana-b988b9b6-pxccw                                        1/1     Running   0          2m53s
default          pod/nfs-client-nfs-subdir-external-provisioner-68f44cd9f4-wjlpv   1/1     Running   0          17h
ingress-nginx    pod/ingress-nginx-controller-6m2vt                                1/1     Running   0          17h
ingress-nginx    pod/ingress-nginx-controller-xkgxl                                1/1     Running   0          17h
kube-system      pod/calico-kube-controllers-684bcfdc59-kmsst                      1/1     Running   0          17h
kube-system      pod/calico-node-dhlnt                                             1/1     Running   0          17h
kube-system      pod/calico-node-r8ktz                                             1/1     Running   0          17h
kube-system      pod/coredns-8474476ff8-9sbwh                                      1/1     Running   0          17h
kube-system      pod/coredns-8474476ff8-fdgcb                                      1/1     Running   0          17h
kube-system      pod/dns-autoscaler-5ffdc7f89d-vskvq                               1/1     Running   0          17h
kube-system      pod/kube-apiserver-node1                                          1/1     Running   0          17h
kube-system      pod/kube-controller-manager-node1                                 1/1     Running   1          17h
kube-system      pod/kube-proxy-hbjz6                                              1/1     Running   0          16h
kube-system      pod/kube-proxy-lfqzt                                              1/1     Running   0          16h
kube-system      pod/kube-scheduler-node1                                          1/1     Running   1          17h
kube-system      pod/kubernetes-dashboard-548847967d-qqngw                         1/1     Running   0          17h
kube-system      pod/kubernetes-metrics-scraper-6d49f96c97-2h7hc                   1/1     Running   0          17h
kube-system      pod/nginx-proxy-node2                                             1/1     Running   0          17h
kube-system      pod/nodelocaldns-64cqs                                            1/1     Running   0          17h
kube-system      pod/nodelocaldns-t5vv6                                            1/1     Running   0          17h
kube-system      pod/registry-proxy-kljvw                                          1/1     Running   0          17h
kube-system      pod/registry-proxy-nz4qk                                          1/1     Running   0          17h
kube-system      pod/registry-xzh9d                                                1/1     Running   0          17h
metallb-system   pod/controller-77c44876d-c92lb                                    1/1     Running   0          17h
metallb-system   pod/speaker-fkjqp                                                 1/1     Running   0          17h
metallb-system   pod/speaker-pqjgt                                                 1/1     Running   0          17h

NAMESPACE     NAME                                TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
default       service/grafana                     LoadBalancer   10.233.1.104    10.10.30.52   80:31116/TCP             2m53s
default       service/kubernetes                  ClusterIP      10.233.0.1      <none>        443/TCP                  17h
kube-system   service/coredns                     ClusterIP      10.233.0.3      <none>        53/UDP,53/TCP,9153/TCP   17h
kube-system   service/dashboard-metrics-scraper   ClusterIP      10.233.35.124   <none>        8000/TCP                 17h
kube-system   service/kubernetes-dashboard        ClusterIP      10.233.32.133   <none>        443/TCP                  17h
kube-system   service/registry                    ClusterIP      10.233.30.221   <none>        5000/TCP                 17h

NAMESPACE        NAME                                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
ingress-nginx    daemonset.apps/ingress-nginx-controller   2         2         2       2            2           kubernetes.io/os=linux   17h
kube-system      daemonset.apps/calico-node                2         2         2       2            2           kubernetes.io/os=linux   17h
kube-system      daemonset.apps/kube-proxy                 2         2         2       2            2           kubernetes.io/os=linux   17h
kube-system      daemonset.apps/nodelocaldns               2         2         2       2            2           kubernetes.io/os=linux   17h
kube-system      daemonset.apps/registry-proxy             2         2         2       2            2           <none>                   17h
metallb-system   daemonset.apps/speaker                    2         2         2       2            2           kubernetes.io/os=linux   17h

NAMESPACE        NAME                                                         READY   UP-TO-DATE   AVAILABLE   AGE
default          deployment.apps/grafana                                      1/1     1            1           2m53s
default          deployment.apps/nfs-client-nfs-subdir-external-provisioner   1/1     1            1           17h
kube-system      deployment.apps/calico-kube-controllers                      1/1     1            1           17h
kube-system      deployment.apps/coredns                                      2/2     2            2           17h
kube-system      deployment.apps/dns-autoscaler                               1/1     1            1           17h
kube-system      deployment.apps/kubernetes-dashboard                         1/1     1            1           17h
kube-system      deployment.apps/kubernetes-metrics-scraper                   1/1     1            1           17h
metallb-system   deployment.apps/controller                                   1/1     1            1           17h

NAMESPACE        NAME                                                                    DESIRED   CURRENT   READY   AGE
default          replicaset.apps/grafana-b988b9b6                                        1         1         1       2m53s
default          replicaset.apps/nfs-client-nfs-subdir-external-provisioner-68f44cd9f4   1         1         1       17h
kube-system      replicaset.apps/calico-kube-controllers-684bcfdc59                      1         1         1       17h
kube-system      replicaset.apps/coredns-8474476ff8                                      2         2         2       17h
kube-system      replicaset.apps/dns-autoscaler-5ffdc7f89d                               1         1         1       17h
kube-system      replicaset.apps/kubernetes-dashboard-548847967d                         1         1         1       17h
kube-system      replicaset.apps/kubernetes-metrics-scraper-6d49f96c97                   1         1         1       17h
kube-system      replicaset.apps/registry                                                1         1         1       17h
metallb-system   replicaset.apps/controller-77c44876d                                    1         1         1       17h

kubectl get ing grafana -o yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/tls-acme: "true"
    meta.helm.sh/release-name: grafana
    meta.helm.sh/release-namespace: default
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  creationTimestamp: "2021-11-11T07:16:12Z"
  generation: 1
  labels:
    app.kubernetes.io/instance: grafana
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: grafana
    app.kubernetes.io/version: 8.2.2
    helm.sh/chart: grafana-6.17.5
  name: grafana
  namespace: default
  resourceVersion: "3137"
  uid: 6c34d3bd-9ab6-42fe-ac1b-7620a9566f62
spec:
  ingressClassName: nginx
  rules:
  - host: grafana.mycluster.org
    http:
      paths:
      - backend:
          service:
            name: ssl-redirect
            port:
              name: use-annotation
        path: /*
        pathType: Prefix
      - backend:
          service:
            name: grafana
            port:
              number: 80
        path: /
        pathType: Prefix
status:
  loadBalancer: {}

Running kubectl get ingressclass returned 'No resources found'.

That's the main reason of your issue.

Why?

When you are specifying ingressClassName: nginx in your Grafana values.yaml file you are setting your Ingress resource to use nginx Ingress class which does not exist.

I replicated your issue using minikube, MetalLB and NGINX Ingress installed via modified deploy.yaml file with commented IngressClass resource + set NGINX Ingress controller name to nginx as in your example. The result was exactly the same - ingressClassName: nginx didn't work (no address), but annotation kubernetes.io/ingress.class: nginx worked.

---

(For the below solution I'm using controller pod name ingress-nginx-controller-86c865f5c4-qwl2b, but in your case it will be different - check it using kubectl get pods -n ingress-nginx command. Also keep in mind it's kind of a workaround - usually ingressClass resource should be installed automatically with a whole installation of NGINX Ingress. I'm presenting this solution to understand why it's not worked for you before, and why it works with NGINX Ingress installed using helm)

In the logs of the Ingress NGINX controller I found (kubectl logs ingress-nginx-controller-86c865f5c4-qwl2b -n ingress-nginx):

"Ignoring ingress because of error while validating ingress class" ingress="default/minimal-ingress" error="no object matching key \"nginx\" in local store"

So it's clearly shown that there is no matching key to nginx controller class - because there is no ingressClass resource which is the "link" between the NGINX Ingress controller and running Ingress resource.

You can verify which name of controller class is bidden to controller by running kubectl get pod ingress-nginx-controller-86c865f5c4-qwl2b -n ingress-nginx -o yaml:

...
spec:
  containers:
  - args:
    - /nginx-ingress-controller
    - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
    - --election-id=ingress-controller-leader
    - --controller-class=k8s.io/nginx
...

Now I will create and apply following Ingress class resource:

apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx
spec:
  controller: k8s.io/nginx

Now in the logs I can see that it's properly configured:

I1115 12:13:42.410384       7 main.go:101] "successfully validated configuration, accepting" ingress="minimal-ingress/default"
I1115 12:13:42.420408       7 store.go:371] "Found valid IngressClass" ingress="default/minimal-ingress" ingressclass="nginx"
I1115 12:13:42.421487       7 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"minimal-ingress", UID:"c708a672-a8dd-45d3-a2ec-f2e2881623ea", APIVersion:"networking.k8s.io/v1", ResourceVersion:"454362", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync

I re-applied the ingress resource definition, I get IP address for Ingress resource.

---

As I said before, instead of using this workaround, I'd suggest installing the NGINX Ingress resource using a solution that automatically installs IngressClass as well. As you have chosen helm chart, it has Ingress Class resource so the problem is gone. Other possible ways to install are here.