AWS

AWS EKS service EXTERNAL-IP of load balancer is Pending

admin

I was working on a staging cluster for my application, it required around 12 load balancers for my services definition. All of 12 looked pretty much the same:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: my-app-api
  name: my-app-api
  namespace: default
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <some aws cert name>
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
spec:
  externalTrafficPolicy: Cluster
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 5001
    - name: https
      port: 443
      protocol: TCP
      targetPort: 5001
  selector:
    app: my-app-api
  sessionAffinity: None
  type: LoadBalancer

After that I went on creating production cluster with the same setup. After I have created it and deployed k8s manifests: deployments, services, I was not able to get LoadBalancer Ingress with kubectl describe service command. I noticed the following picture:

NAME          TYPE           CLUSTER-IP  EXTERNAL-IP                         PORT(S)                      AGE
<some-name>   LoadBalancer   <some_ip>   ****.us-west-1.elb.amazonaws.com    80:30339/TCP,443:32754/TCP   1m
<some-name>   LoadBalancer   <some_ip>   ****.us-west-1.elb.amazonaws.com    80:31538/TCP,443:32061/TCP   1m
<some-name>   LoadBalancer   <some_ip>   ****.us-west-1.elb.amazonaws.com    80:30976/TCP,443:31323/TCP   1m
<some-name>   LoadBalancer   <some_ip>   ****.us-west-1.elb.amazonaws.com    80:30288/TCP,443:32073/TCP   1m
<some-name>   LoadBalancer   <some_ip>   ****.us-west-1.elb.amazonaws.com    80:32270/TCP,443:31159/TCP   1m
<some-name>   LoadBalancer   <some_ip>   ****.us-west-1.elb.amazonaws.com    80:31966/TCP,443:30944/TCP   1m
kubernetes    ClusterIP      <some_ip>   <none>                              443/TCP                      1m
<some-name>   LoadBalancer   <some_ip>   PENDING                             80:31901/TCP,443:30444/TCP   1m
<some-name>   LoadBalancer   <some_ip>   PENDING                             80:31510/TCP,443:30393/TCP   1m
<some-name>   LoadBalancer   <some_ip>   PENDING                             80:32613/TCP,443:32616/TCP   1m
<some-name>   LoadBalancer   <some_ip>   PENDING                             80:32069/TCP,443:30320/TCP   1m
<some-name>   LoadBalancer   <some_ip>   PENDING                             80:31667/TCP,443:32194/TCP   1m
<some-name>   LoadBalancer   <some_ip>   PENDING                             80:31943/TCP,443:32081/TCP   1m

Answer

After troubleshooting the reason of above behaviour, I have made the following conclusions about LoadBalancers(LB):

  1. service.beta.kubernetes.io/aws-load-balancer-ssl-cert directly depends on AWS load balancer, and if certificate is not signed in a region where LB is created, LB will not be added to the k8s cluster.
  2. My problem was because k8s cluster could not automatically provision AWS LB for newly created service, as default LB limit for single region in AWS account is 20

I requested quota increase LB limit from AWS, but since it took more time, I moved my production cluster to a different AWS region. After that LBs created as expected and I could get my ingresses.