AWS Load Balancer Controller successfully creates ALB when Ingress is deployed, but unable to get DNS Name in CDK code

Question 1

I originally posted this question as an issue on the GitHub project for the AWS Load Balancer Controller here: https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/2069.

I'm seeing some odd behavior that I can't trace or explain when trying to get the loadBalacnerDnsName from an ALB created by the controller. I'm using v2.2.0 of the AWS Load Balancer Controller in a CDK project. The ingress that I deploy triggers the provisioning of an ALB, and that ALB can connect to my K8s workloads running in EKS.

Here's my problem: I'm trying to automate the creation of a Route53 A Record that points to the loadBalancerDnsName of the load balancer, but the loadBalancerDnsName that I get in my CDK script is not the same as the loadBalancerDnsName that shows up in the AWS console once my stack has finished deploying. The value in the console is correct and I can get a response from that URL. My CDK script outputs the value of the DnsName as a CfnOutput value, but that URL does not point to anything.

In CDK, I have tried to use KubernetesObjectValue to get the DNS name from the load balancer. This isn't working (see this related issue: https://github.com/aws/aws-cdk/issues/14933), so I'm trying to lookup the Load Balancer with CDK's .fromLookup and using a tag that I added through my ingress annotation:

    const alb = elbv2.ApplicationLoadBalancer.fromLookup(this, 'appAlb', {
      loadBalancerTags: {
        Environment: 'test',
      },
    });

Here's the project in which I'm running into this issue: https://github.com/briancaffey/django-cdk

Here are some relevant files:

Installing the AWS Load Balancer Controller with CDK: https://github.com/briancaffey/django-cdk/blob/main/src/eks/awslbc/index.ts

Ingress object definition: https://github.com/briancaffey/django-cdk/tree/main/src/eks/resources/ingress

ALB fromLookup method that is generating the wrong DnsName: https://github.com/briancaffey/django-cdk/blob/main/src/django-eks.ts#L297

Does anyone have any ideas about what could be causing this or how I can debug? If there is any other information that would be helpful to collect in my debugging process, please let me know.

I'm not sure if this is possible, but it seems like the Load Balancer might be created, assigned a DnsName, and then maybe fails provisioning at some point and tries to create another ALB that with another DnsName that is successful.

Update: Here are logs from the AWS Load Balancer Controller deployment:

~/git/github/django-cdk$ kubectl logs deployment/goeksstackdjangoekssamplealbingresscontroller48a16415-aws-load -n kube-system

Found 2 pods, using pod/goeksstackdjangoekssamplealbingresscontroller48a16415-aws-c6d8h
{"level":"info","ts":1623379830.094481,"msg":"version","GitVersion":"v2.2.0","GitCommit":"68c417a7ea37ff153f053d9ffef1cc5c70d7e211","BuildDate":"2021-05-14T21:49:05+0000"}
{"level":"info","ts":1623379830.1275837,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1623379830.1310954,"logger":"setup","msg":"adding health check for controller"}
{"level":"info","ts":1623379830.1312613,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/mutate-v1-pod"}
{"level":"info","ts":1623379830.131368,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/mutate-elbv2-k8s-aws-v1beta1-targetgroupbinding"}
{"level":"info","ts":1623379830.131444,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validate-elbv2-k8s-aws-v1beta1-targetgroupbinding"}
{"level":"info","ts":1623379830.1315207,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validate-networking-v1beta1-ingress"}
{"level":"info","ts":1623379830.1316643,"logger":"setup","msg":"starting podInfo repo"}
I0611 02:50:32.131767       1 leaderelection.go:242] attempting to acquire leader lease  kube-system/aws-load-balancer-controller-leader...
{"level":"info","ts":1623379832.1318467,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
I0611 02:50:32.150030       1 leaderelection.go:252] successfully acquired lease kube-system/aws-load-balancer-controller-leader
{"level":"info","ts":1623379832.2321155,"logger":"controller-runtime.webhook.webhooks","msg":"starting webhook server"}
{"level":"info","ts":1623379832.2321231,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"elbv2.k8s.aws","reconcilerKind":"TargetGroupBinding","controller":"targetGroupBinding","source":"kind source: /, Kind="}
{"level":"info","ts":1623379832.2321854,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"elbv2.k8s.aws","reconcilerKind":"TargetGroupBinding","controller":"targetGroupBinding","source":"kind source: /, Kind="}
{"level":"info","ts":1623379832.2322075,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"elbv2.k8s.aws","reconcilerKind":"TargetGroupBinding","controller":"targetGroupBinding","source":"kind source: /, Kind="}
{"level":"info","ts":1623379832.2323742,"logger":"controller","msg":"Starting EventSource","controller":"ingress","source":"channel source: 0xc00053a820"}
{"level":"info","ts":1623379832.2324347,"logger":"controller","msg":"Starting EventSource","controller":"ingress","source":"channel source: 0xc00053a870"}
{"level":"info","ts":1623379832.2324626,"logger":"controller","msg":"Starting EventSource","controller":"ingress","source":"kind source: /, Kind="}
{"level":"info","ts":1623379832.2324827,"logger":"controller","msg":"Starting EventSource","controller":"ingress","source":"kind source: /, Kind="}
{"level":"info","ts":1623379832.2325017,"logger":"controller","msg":"Starting EventSource","controller":"ingress","source":"kind source: /, Kind="}
{"level":"info","ts":1623379832.232624,"logger":"controller","msg":"Starting EventSource","controller":"service","source":"kind source: /, Kind="}
{"level":"info","ts":1623379832.2326622,"logger":"controller","msg":"Starting Controller","controller":"service"}
{"level":"info","ts":1623379832.232741,"logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":1623379832.2328286,"logger":"controller-runtime.webhook","msg":"serving webhook server","host":"","port":9443}
{"level":"info","ts":1623379832.2332337,"logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"}
{"level":"info","ts":1623379832.3324778,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"elbv2.k8s.aws","reconcilerKind":"TargetGroupBinding","controller":"targetGroupBinding","source":"kind source: /, Kind="}
{"level":"info","ts":1623379832.3327508,"logger":"controller","msg":"Starting workers","controller":"service","worker count":3}
{"level":"info","ts":1623379832.332836,"logger":"controller","msg":"Starting EventSource","controller":"ingress","source":"channel source: 0xc00053a8c0"}
{"level":"info","ts":1623379832.3328674,"logger":"controller","msg":"Starting EventSource","controller":"ingress","source":"kind source: /, Kind="}
{"level":"info","ts":1623379832.4330835,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"elbv2.k8s.aws","reconcilerKind":"TargetGroupBinding","controller":"targetGroupBinding"}
{"level":"info","ts":1623379832.4333706,"logger":"controller","msg":"Starting workers","reconcilerGroup":"elbv2.k8s.aws","reconcilerKind":"TargetGroupBinding","controller":"targetGroupBinding","worker count":3}
{"level":"info","ts":1623379832.433311,"logger":"controller","msg":"Starting EventSource","controller":"ingress","source":"kind source: /, Kind="}
{"level":"info","ts":1623379832.4334028,"logger":"controller","msg":"Starting Controller","controller":"ingress"}
{"level":"info","ts":1623379832.433451,"logger":"controller","msg":"Starting workers","controller":"ingress","worker count":3}
{"level":"info","ts":1623379851.8822243,"logger":"controllers.ingress","msg":"successfully built model","model":"{\"id\":\"app/app-ingress\",\"resources\":{\"AWS::EC2::SecurityGroup\":{\"ManagedLBSecurityGroup\":{\"spec\":{\"groupName\":\"k8s-app-appingre-64b5836d2b\",\"description\":\"[k8s] Managed SecurityGroup for LoadBalancer\",\"tags\":{\"Environment\":\"test\"},\"ingress\":[{\"ipProtocol\":\"tcp\",\"fromPort\":80,\"toPort\":80,\"ipRanges\":[{\"cidrIP\":\"0.0.0.0/0\"}]}]}}},\"AWS::ElasticLoadBalancingV2::Listener\":{\"80\":{\"spec\":{\"loadBalancerARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::LoadBalancer/LoadBalancer/status/loadBalancerARN\"},\"port\":80,\"protocol\":\"HTTP\",\"defaultActions\":[{\"type\":\"fixed-response\",\"fixedResponseConfig\":{\"contentType\":\"text/plain\",\"statusCode\":\"404\"}}],\"tags\":{\"Environment\":\"test\"}}}},\"AWS::ElasticLoadBalancingV2::ListenerRule\":{\"80:1\":{\"spec\":{\"listenerARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::Listener/80/status/listenerARN\"},\"priority\":1,\"actions\":[{\"type\":\"forward\",\"forwardConfig\":{\"targetGroups\":[{\"targetGroupARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/app/app-ingress-api-http:80/status/targetGroupARN\"}}]}}],\"conditions\":[{\"field\":\"path-pattern\",\"pathPatternConfig\":{\"values\":[\"/*\"]}}],\"tags\":{\"Environment\":\"test\"}}}},\"AWS::ElasticLoadBalancingV2::LoadBalancer\":{\"LoadBalancer\":{\"spec\":{\"name\":\"k8s-app-appingre-df49e963f5\",\"type\":\"application\",\"scheme\":\"internet-facing\",\"ipAddressType\":\"ipv4\",\"subnetMapping\":[{\"subnetID\":\"subnet-0257383f56bbc4810\"},{\"subnetID\":\"subnet-0b8b2e282788f64bc\"}],\"securityGroups\":[{\"$ref\":\"#/resources/AWS::EC2::SecurityGroup/ManagedLBSecurityGroup/status/groupID\"}],\"tags\":{\"Environment\":\"test\"}}}},\"AWS::ElasticLoadBalancingV2::TargetGroup\":{\"app/app-ingress-api-http:80\":{\"spec\":{\"name\":\"k8s-app-apihttp-ee37d33f98\",\"targetType\":\"instance\",\"port\":30867,\"protocol\":\"HTTP\",\"protocolVersion\":\"HTTP1\",\"healthCheckConfig\":{\"port\":\"traffic-port\",\"protocol\":\"HTTP\",\"path\":\"/\",\"matcher\":{\"httpCode\":\"200\"},\"intervalSeconds\":15,\"timeoutSeconds\":5,\"healthyThresholdCount\":2,\"unhealthyThresholdCount\":2},\"tags\":{\"Environment\":\"test\"}}}},\"K8S::ElasticLoadBalancingV2::TargetGroupBinding\":{\"app/app-ingress-api-http:80\":{\"spec\":{\"template\":{\"metadata\":{\"name\":\"k8s-app-apihttp-ee37d33f98\",\"namespace\":\"app\",\"creationTimestamp\":null},\"spec\":{\"targetGroupARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/app/app-ingress-api-http:80/status/targetGroupARN\"},\"targetType\":\"instance\",\"serviceRef\":{\"name\":\"api-http\",\"port\":80},\"networking\":{\"ingress\":[{\"from\":[{\"securityGroup\":{\"groupID\":{\"$ref\":\"#/resources/AWS::EC2::SecurityGroup/ManagedLBSecurityGroup/status/groupID\"}}}],\"ports\":[{\"protocol\":\"TCP\"}]}]}}}}}}}}"}
{"level":"info","ts":1623379852.083325,"logger":"controllers.ingress","msg":"creating securityGroup","resourceID":"ManagedLBSecurityGroup"}
{"level":"info","ts":1623379852.2566178,"logger":"controllers.ingress","msg":"created securityGroup","resourceID":"ManagedLBSecurityGroup","securityGroupID":"sg-080bffd697dc0ab27"}
{"level":"info","ts":1623379852.4109924,"msg":"authorizing securityGroup ingress","securityGroupID":"sg-080bffd697dc0ab27","permission":[{"FromPort":80,"IpProtocol":"tcp","IpRanges":[{"CidrIp":"0.0.0.0/0","Description":""}],"Ipv6Ranges":null,"PrefixListIds":null,"ToPort":80,"UserIdGroupPairs":null}]}
{"level":"info","ts":1623379852.5575335,"msg":"authorized securityGroup ingress","securityGroupID":"sg-080bffd697dc0ab27"}
{"level":"info","ts":1623379852.6474898,"logger":"controllers.ingress","msg":"creating targetGroup","stackID":"app/app-ingress","resourceID":"app/app-ingress-api-http:80"}
{"level":"info","ts":1623379852.9693868,"logger":"controllers.ingress","msg":"created targetGroup","stackID":"app/app-ingress","resourceID":"app/app-ingress-api-http:80","arn":"arn:aws:elasticloadbalancing:us-east-1:111111111111:targetgroup/k8s-app-apihttp-ee37d33f98/8e8cf427880a005f"}
{"level":"info","ts":1623379853.0108032,"logger":"controllers.ingress","msg":"creating loadBalancer","stackID":"app/app-ingress","resourceID":"LoadBalancer"}
{"level":"info","ts":1623379853.6124952,"logger":"controllers.ingress","msg":"created loadBalancer","stackID":"app/app-ingress","resourceID":"LoadBalancer","arn":"arn:aws:elasticloadbalancing:us-east-1:111111111111:loadbalancer/app/k8s-app-appingre-df49e963f5/e1e08ff54e34da78"}
{"level":"info","ts":1623379853.6379986,"logger":"controllers.ingress","msg":"creating listener","stackID":"app/app-ingress","resourceID":"80"}
{"level":"info","ts":1623379853.7017045,"logger":"controllers.ingress","msg":"created listener","stackID":"app/app-ingress","resourceID":"80","arn":"arn:aws:elasticloadbalancing:us-east-1:111111111111:listener/app/k8s-app-appingre-df49e963f5/e1e08ff54e34da78/7896e4ee12889f1d"}
{"level":"info","ts":1623379853.7323081,"logger":"controllers.ingress","msg":"creating listener rule","stackID":"app/app-ingress","resourceID":"80:1"}
{"level":"info","ts":1623379853.7993383,"logger":"controllers.ingress","msg":"created listener rule","stackID":"app/app-ingress","resourceID":"80:1","arn":"arn:aws:elasticloadbalancing:us-east-1:111111111111:listener-rule/app/k8s-app-appingre-df49e963f5/e1e08ff54e34da78/7896e4ee12889f1d/2fc270d382a475d1"}
{"level":"info","ts":1623379853.7995286,"logger":"controllers.ingress","msg":"creating targetGroupBinding","stackID":"app/app-ingress","resourceID":"app/app-ingress-api-http:80"}
{"level":"info","ts":1623379853.836185,"logger":"controllers.ingress","msg":"created targetGroupBinding","stackID":"app/app-ingress","resourceID":"app/app-ingress-api-http:80","targetGroupBinding":{"namespace":"app","name":"k8s-app-apihttp-ee37d33f98"}}
{"level":"info","ts":1623379853.989902,"logger":"controllers.ingress","msg":"successfully deployed model","ingressGroup":"app/app-ingress"}
{"level":"info","ts":1623379854.0531545,"msg":"authorizing securityGroup ingress","securityGroupID":"sg-03aa126eb1aaeacc5","permission":[{"FromPort":0,"IpProtocol":"tcp","IpRanges":null,"Ipv6Ranges":null,"PrefixListIds":null,"ToPort":65535,"UserIdGroupPairs":[{"Description":"elbv2.k8s.aws/targetGroupBinding=shared","GroupId":"sg-080bffd697dc0ab27","GroupName":null,"PeeringStatus":null,"UserId":null,"VpcId":null,"VpcPeeringConnectionId":null}]}]}
{"level":"info","ts":1623379854.1990266,"msg":"authorized securityGroup ingress","securityGroupID":"sg-03aa126eb1aaeacc5"}
{"level":"info","ts":1623379854.3660266,"msg":"registering targets","arn":"arn:aws:elasticloadbalancing:us-east-1:111111111111:targetgroup/k8s-app-apihttp-ee37d33f98/8e8cf427880a005f","targets":[{"AvailabilityZone":null,"Id":"i-08b9e38bcf885a07a","Port":30867},{"AvailabilityZone":null,"Id":"i-0cdcd5bd12476c990","Port":30867}]}
{"level":"info","ts":1623379854.5757735,"msg":"registered targets","arn":"arn:aws:elasticloadbalancing:us-east-1:111111111111:targetgroup/k8s-app-apihttp-ee37d33f98/8e8cf427880a005f"}

Here are the logs for the other pod in the AWS Load Balancer Controller deployment:

~/git/github/django-cdk$ kubectl logs -n kube-system goeksstackdjangoekssamplealbingresscontroller48a16415-aws-q9nmg
{"level":"info","ts":1623379840.7888832,"msg":"version","GitVersion":"v2.2.0","GitCommit":"68c417a7ea37ff153f053d9ffef1cc5c70d7e211","BuildDate":"2021-05-14T21:49:05+0000"}
{"level":"info","ts":1623379840.8260994,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1623379840.8316746,"logger":"setup","msg":"adding health check for controller"}
{"level":"info","ts":1623379840.8317719,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/mutate-v1-pod"}
{"level":"info","ts":1623379840.8318048,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/mutate-elbv2-k8s-aws-v1beta1-targetgroupbinding"}
{"level":"info","ts":1623379840.83182,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validate-elbv2-k8s-aws-v1beta1-targetgroupbinding"}
{"level":"info","ts":1623379840.831849,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validate-networking-v1beta1-ingress"}
{"level":"info","ts":1623379840.8319223,"logger":"setup","msg":"starting podInfo repo"}
I0611 02:50:42.832101       1 leaderelection.go:242] attempting to acquire leader lease  kube-system/aws-load-balancer-controller-leader...
{"level":"info","ts":1623379842.8324547,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1623379842.9327362,"logger":"controller-runtime.webhook.webhooks","msg":"starting webhook server"}
{"level":"info","ts":1623379842.9332633,"logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":1623379842.9333658,"logger":"controller-runtime.webhook","msg":"serving webhook server","host":"","port":9443}
{"level":"info","ts":1623379842.9334989,"logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"}

I replaced my account ID with 111111111111.

Here is the ALB DNS name from my CDK Stack's CfnOutput:

k8s-app-appingre-a5bb1f9208-1217069225.us-east-1.elb.amazonaws.com

and here is the DNS name for the ALB from the EC2 > Load Balancers console:

k8s-app-appingre-df49e963f5-842078657.us-east-1.elb.amazonaws.com

Question 2

Use this "Ingress.yaml" as example:

# Annotations Reference:  https://kubernetes-sigs.github.io/aws-alb-ingress-controller/guide/ingress/annotation/
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: eks-microservices-demo
  labels:
    app: usermgmt-restapp
  annotations:
    # Ingress Core Settings  
    kubernetes.io/ingress.class: "alb"
    alb.ingress.kubernetes.io/scheme: internet-facing
    # Health Check Settings
    alb.ingress.kubernetes.io/healthcheck-protocol: HTTP 
    alb.ingress.kubernetes.io/healthcheck-port: traffic-port
    alb.ingress.kubernetes.io/healthcheck-interval-seconds: '15'
    alb.ingress.kubernetes.io/healthcheck-timeout-seconds: '5'
    alb.ingress.kubernetes.io/success-codes: '200'
    alb.ingress.kubernetes.io/healthy-threshold-count: '2'
    alb.ingress.kubernetes.io/unhealthy-threshold-count: '2'
    ## SSL Settings
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:ddddddddddddd
    #alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-1-2017-01 #Optional (Picks default if not used)    
    # SSL Redirect Setting
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'   
    # External DNS - For creating a Record Set in Route53
    external-dns.alpha.kubernetes.io/hostname: users.skycomposer.net       
spec:
  rules:
    - http:
        paths:
          - path: /* # SSL Redirect Setting
            backend:
              serviceName: ssl-redirect
              servicePort: use-annotation            
          - path: /*
            backend:
              serviceName: usermgmt-restapp-nodeport-service
              servicePort: 8095                                   
# Important Note-1: In path based routing order is very important, if we are going to use  "/*", try to use it at the end of all rules.

P.S. Warning! It will only work with managed AWS EKS cluster, or any other cluster, which supports ALB Ingress Controller.

P.P.S. With K3S Kubernetes cluster, I use Traefik Ingress Controller and have to manually create Route 53 CNAME Record with the name of domain and the value of Load Balancer DNS Name. Unfortunately, I don't know how to automate this process.

The full source code example of AWS K3S Kuberneter Cluster with Traefik Ingress Controller, with step-by-step instructions, can be found here: https://github.com/skyglass/customer-management-keycloak

AWS Load Balancer Controller successfully creates ALB when Ingress is deployed, but unable to get DNS Name in CDK code

K8SOQ

Answer a question

Answers

所有评论(0)

K8SOQ