Internal and External AWS LB with external-dns

Traefik Traefik v3 (latest)
1

Hi,
I want to setup EKS with

  • Internal LB that handles traffic from lambdas and other services to a Pod
  • External LB for customers

I installed traefik via helm-chart and added an extraObject ingressClass:

  • traefik - as default, set for external LB
  • traefik-internal for internal LB
ingressClass:
    enabled: true
    isDefaultClass: true
    name: "${ingress_class}"
extraObjects:
  - apiVersion: networking.k8s.io/v1
    kind: IngressClass
    metadata:
      name: "${ingress_class}-internal"
      annotations:
        ingressclass.kubernetes.io/is-default-class: "false"
    spec:
      controller: traefik.io/ingress-controller

Services, providers, ports:

service:
    annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: nlb
        service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "${ssl_cert}"
        service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
        service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
        service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    type: LoadBalancer
    externalTrafficPolicy: Local
    additionalServices:
        internal:
            type: LoadBalancer
            annotations:
                service.beta.kubernetes.io/aws-load-balancer-internal: "true"
                service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
                service.beta.kubernetes.io/aws-load-balancer-type: nlb
                service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"

providers:
    kubernetesCRD:
        enabled: true
        ingressClass: "${ingress_class}"
    kubernetesIngress:
        enabled: true
        ingressClass: "${ingress_class}"
        publishedService: 
            enabled: true


ports:
  websecure:
    tls:
      enabled: false
    appProtocol: HTTP
  web-int:
    port: 8085
    expose:
      default: false
      internal: true
    exposedPort: 8085
  websecure-int:
    port: 8445
    tls:
      enabled: false
    expose:
      default: false
      internal: true
    appProtocol: HTTP
    exposedPort: 8445

Now what I want is to setup 2 ingresses for my application to run internal and externally

External:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/hostname: example.host
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.middlewares: argo-app-strip-api@kubernetescrd
  labels:
    argocd.argoproj.io/instance: example
  name: example
  namespace: argo-app
spec:
  ingressClassName: traefik
  rules:
    - host: example.host
      http:
        paths:
          - backend:
              service:
                name: example-backend
                port:
                  number: 8000
            path: /api/
            pathType: Prefix
          - backend:
              service:
                name: example-frontend
                port:
                  number: 80
            path: /
            pathType: Prefix

And internal:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/hostname: example.intern
    traefik.ingress.kubernetes.io/router.entrypoints: web-int
    traefik.ingress.kubernetes.io/router.middlewares: argo-app-strip-api@kubernetescrd
  labels:
    argocd.argoproj.io/instance: example
  name: example-internal-ingress
  namespace: argo-app
spec:
  ingressClassName: traefik-internal
  rules:
    - host: example.intern
      http:
        paths:
          - backend:
              service:
                name: example-backend
                port:
                  number: 8000
            path: /api/
            pathType: Prefix
          - backend:
              service:
                name: example-frontend
                port:
                  number: 80
            path: /
            pathType: Prefix

The problem is though, that external DNS seems to use the internal LB for every ingress, ignoring router.entrypoints and ingressClassName → Therefore the external LB is set by externalDNS for every ingress which I can see in the .intern hostedZone which shows it linking to the external LB.
How can I solve this? And is this even the correct approach?

Thanks!

2

The relatively easy fix is, to deploy 2 traefik controller side-by-side

Internal

ingressClass:
    enabled: true
    isDefaultClass: false
    name: "${ingress_class}"

ingressRoute:
    dashboard:
        enabled: true

service:
    annotations:
        service.beta.kubernetes.io/aws-load-balancer-internal: "true"
        service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
        service.beta.kubernetes.io/aws-load-balancer-type: nlb
        service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    type: LoadBalancer
    externalTrafficPolicy: Local

providers:
    kubernetesCRD:
        enabled: true
        ingressClass: "${ingress_class}"
    kubernetesIngress:
        enabled: true
        ingressClass: "${ingress_class}"
        publishedService: 
            enabled: true
ports:
  websecure:
    tls:
      enabled: false
    appProtocol: HTTP

External

nodeSelector: 
    eks.amazonaws.com/nodegroup: ${node_group_name}
ingressClass:
    enabled: true
    isDefaultClass: true
    name: "${ingress_class}"

service:
    annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: nlb
        service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "${ssl_cert}"
        service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
        service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
        service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    type: LoadBalancer
    externalTrafficPolicy: Local

providers:
    kubernetesCRD:
        enabled: true
        ingressClass: "${ingress_class}"
    kubernetesIngress:
        enabled: true
        ingressClass: "${ingress_class}"
        publishedService: 
            enabled: true
ports:
  websecure:
    tls:
      enabled: false
    appProtocol: HTTP

Setting the names differently (in Terraform):


resource "helm_release" "traefik" {
  name       = "traefik"
  repository = "https://traefik.github.io/charts"
  chart      = "traefik"
  namespace  = "traefik"
  version    = var.traefik_chart_version

  create_namespace = true
  values = [templatefile("${path.module}/tf-templates/${var.target_branch}-traefik.tftpl", {
    ingress_class = var.traefik_ingress_class
  })]
}


variable "traefik_ingress_class" {
  description = "value of traefik ingress class"
  type        = string
  default     = "traefik"
}

resource "helm_release" "traefik_internal" {
  depends_on = [
    aws_eks_node_group.rcs_eks_infrastructure_node_group,
    helm_release.cluster_autoscaler
  ]
  name       = var.traefik_ingress_class_internal
  repository = "https://traefik.github.io/charts"
  chart      = "traefik"
  namespace  = "traefik"
  version    = var.traefik_chart_version

  create_namespace = true
  values = [templatefile("${path.module}/tf-templates/${var.target_branch}-traefik-internal.tftpl", {
    ingress_class   = var.traefik_ingress_class_internal
  })]
}


variable "traefik_ingress_class_internal" {
  description = "value of traefik ingress class"
  type        = string
  default     = "traefik-internal"
}

Now setting the ingresses just to use web and websecure but using the internal ingressClass will make them available internally.

ingress:
  enabled: true
  className: traefik
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    external-dns.alpha.kubernetes.io/hostname: example.host
  hostname: example.host
ingressInternal:
  enabled: true
  className: traefik-internal
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: web
    external-dns.alpha.kubernetes.io/hostname: example.intern
  hostname: example.intern

Fyi: We are using external-dns to automatically add domains to their hosted zones.

Works :smiley: