Intermittent Response and Routing from Traefik in AWS ECS/Fargate

Traefik Traefik v2
,
1

I am trying to configure traefik proxy to run within AWS Elastic Container Service using Fargate containers.

I am using the Docker Compose ECS integration to create and configure the cluster, tasks, and services.

There is a network load balancer configured to serve the TLS certificate and terminate TLS encryption, and then Traefik routes based on rules.

Once all the services are up, this setup works, but after some time the Traefik dashboard and any services are unreachable. I haven't been able to find any useful information within the logs.

The NLB is placed in a public subnet while any services are placed within private subnets.

Route53 -- HTTPS --> NetworkLoadBalancer -- HTTP --> Traefik Instance

x-aws-vpc: "vpc-xxxx"
x-aws-loadbalancer: "LOADBALANCER-ARN"

services:
  traefik:
    image: traefik:2.7
    ports:
      - target: 80
        published: 80
        protocol: tcp
        x-aws-protocol: tcp
    command:
      ## API Settings - https://docs.traefik.io/operations/api/, endpoints - https://docs.traefik.io/operations/api/#endpoints ##
      - "--api.insecure=true"
      - "--api.dashboard=true"
      - "--api.debug=true"
      ## Log Settings (options: ERROR, DEBUG, PANIC, FATAL, WARN, INFO) - https://docs.traefik.io/observability/logs/ ##
      - "--log.level=INFO"
      ## Provider Settings - https://docs.traefik.io/providers/docker/#provider-configuration ##
      - "--providers.ecs=true"
      - "--providers.ecs.region=us-east-2"
      - "--providers.ecs.accessKeyID=..."
      - "--providers.ecs.secretAccessKey=..."
      - "--providers.ecs.autoDiscoverClusters=true"
      - "--providers.ecs.refreshSeconds=15"
      - "--providers.ecs.exposedbydefault=false" 

      ## Entrypoints Settings - https://docs.traefik.io/routing/entrypoints/#configuration ##
      - "--entrypoints.web.address=:80"
      - "--entryPoints.web.forwardedHeaders.insecure" # I do not like this...

    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.rule=Host(`traefik.example.com`)"
      - "traefik.http.routers.traefik.service=api@internal" # <== Enabling the api to be a service to access
      - "traefik.http.services.traefik.loadbalancer.server.port=8080"
      
    networks:
      - frontend
      - backend
    deploy:
      replicas: 1
      x-aws-autoscaling: 
        min: 1
        max: 6
        cpu: 50
    x-aws-role:
      Version: "2012-10-17"
      Statement:
        - Effect: "Allow"
          Action:
            - "ecs:ListClusters"
            - "ecs:DescribeClusters"
            - "ecs:ListTasks"
            - "ecs:DescribeTasks"
            - "ecs:DescribeContainerInstances"
            - "ecs:DescribeTaskDefinition"
            - "ec2:DescribeInstances"
            - "ecr:BatchCheckLayerAvailability"
            - "ecr:BatchGetImage"
            - "ecr:GetDownloadUrlForLayer"
            - "ecr:GetAuthorizationToken"
            - "logs:CreateLogStream"
            - "logs:PutLogEvents"
            - "kms:Decrypt"
            - "secretsmanager:GetSecretValue"
          Resource:
            - "*"  
  nginx:
    image: nginx:latest
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.nginx.rule=Host(`nginx.example.com`)"
      - "traefik.http.routers.nginx.entrypoints=web"
      - "traefik.http.services.nginx.loadbalancer.server.port=80"
    networks:
      - backend

networks:
  frontend:
    external: true
    name: sg-for-the-frontend
  backend:
    external: true
    name: sg-for-the-backend


x-aws-cloudformation:
  Resources:
    TraefikTCP80Listener:
      Properties:
        Certificates:
          - CertificateArn: "certificateARN"
        Protocol: TLS
        Port: 443
    TraefikService:
      Properties:
        NetworkConfiguration:
          AwsvpcConfiguration:
            Subnets:
              - subnet-private1
              - subnet-private2
              - subnet-private3
    NginxService:
      Properties:
        NetworkConfiguration:
          AwsvpcConfiguration:
            Subnets:
              - subnet-private1
              - subnet-private2
              - subnet-private3