Traefik Not Working

Traefik Traefik v2
, , ,
1

Running docker swarm mode on a single node hetzner vps. Firewall is correctly configured to allow traffic on necessary ports, 22, 80, 443, 9000 for now to view portainer to see what's going wrong, 2377 for network communication. Read the docs of traefik, swarm, portainer, cover to cover. Looks like my configurations are correct but still does not work.

docker stack file:

version: "3.8"
services:
  postgres:
    image: pgvector/pgvector:pg16
    environment:
      POSTGRES_DB:
      POSTGRES_USER:
      POSTGRES_PASSWORD:
    volumes:
      - postgres_data:/var/lib/postgresql/data
    networks:
      - traefik-network
    deploy:
      # placement:
        # constraints: [node.role == worker]
      # labels:
      #   - "traefik.enable=true"
      #   - "traefik.http.routers.postgres.rule=Host(`db.domain.com`)"
      #   - "traefik.http.routers.postgres.entrypoints=web"
      #   - "traefik.http.routers.postgres.entrypoints=websecure"
      #   - "traefik.http.routers.postgres.tls.certresolver=traefikresolver"
      #   - "traefik.http.services.postgres.loadbalancer.server.port=5432"
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER}"]
      interval: 10s
      timeout: 5s
      retries: 5

  django:
    image: name/image:v1.0.0
    command: gunicorn config.asgi:application --worker-class uvicorn.workers.UvicornWorker --workers 2 --bind 0.0.0.0:8000
    networks:
      - traefik-network
    deploy:
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.django.rule=Host(`api.domain.com`)"
        - "traefik.http.routers.django.entrypoints=web"
        - "traefik.http.routers.django.entrypoints=websecure"
        - "traefik.http.routers.django.tls.certresolver=traefikresolver"
        - "traefik.http.services.django.loadbalancer.server.port=8000"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://0.0.0.0:8000/healthcheck"]
      interval: 30s
      timeout: 5s
      retries: 5
  
  traefik:
    image: traefik:v2.11
    command:
      # https://doc.traefik.io/traefik/reference/static-configuration/cli/
      # api
      - --api.dashboard=true
      - --api.insecure=true
      - --log.level=DEBUG
      - --providers.docker=true
      - --providers.docker.endpoint=unix:///var/run/docker.sock
      - --providers.docker.swarmmode=true
      - --providers.docker.exposedbydefault=false  # Don't expose every service by default
      - --providers.docker.network=traefik-network
      # - --providers.docker.watch=true
      - --entrypoints.web
      - --entrypoints.web.address=:80  # Define HTTP entry point
      - --entrypoints.websecure
      - --entrypoints.websecure.address=:443  # Define HTTPS entry point
      - --entrypoints.websecure.http.tls.certresolver=traefikresolver
      - --certificatesresolvers.traefikresolver
      - --certificatesresolvers.traefikresolver.acme.httpchallenge=true
      - --certificatesresolvers.traefikresolver.acme.httpchallenge.entrypoint=web
      - --certificatesresolvers.traefikresolver.acme.email=email@gmail.com
      - --certificatesresolvers.traefikresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory # staging
      - --certificatesresolvers.traefikresolver.acme.storage=/letsencrypt/staging/acme.json # staging
      # - --certificatesresolvers.traefikresolver.acme.caserver=https://acme-v02.api.letsencrypt.org/directory # prod
      # - --certificatesresolvers.traefikresolver.acme.storage=/letsencrypt/acme.json # prod
    ports:
      - "80:80"  # Expose HTTP
      - "443:443"  # Expose HTTPS
      - "8080:8080"  # Expose Traefik Dashboard
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"  # Allow Traefik to listen to Docker events
      - "~/mnt/data/traefik/acme.json:/letsencrypt/acme.json"
      - "~/mnt/data/traefik/staging/acme.json:/letsencrypt/staging/acme.json"
    deploy:
      placement:
        constraints:
          - node.role == manager
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.api.rule=Host(`traefik.domain.com`)"
        - "traefik.http.routers.api.service=api@internal"
        - "traefik.http.routers.api.middlewares=authtraefik"
        - "traefik.http.middlewares.authtraefik.basicauth.users=admin:password"
        - "traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)"
        - "traefik.http.routers.http-catchall.entrypoints=web"
        - "traefik.http.routers.http-catchall.middlewares=redirect-to-https"
        - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
    networks:
      - traefik-network
    healthcheck:
      test: ["CMD", "traefik", "healthcheck", "--ping"]
      interval: 15s
      timeout: 3s
      retries: 3
      start_period: 120s

  portainer:
    image: portainer/portainer-ce
    ports:
      - "9000:9000"
      - "9001:9001"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
      - "~/mnt/data/portainer:/data"
    command: -H unix:///var/run/docker.sock
    networks:
      - traefik-network
    deploy:
      # placement:
        # constraints: [node.role == manager]
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.portainer.rule=Host(`portainer.domain.com`)"
        - "traefik.http.routers.portainer.entrypoints=websecure"
        - "traefik.http.routers.portainer.tls=true"
        - "traefik.http.routers.portainer.tls.certresolver=traefikresolver"
        - "traefik.http.services.portainer.loadbalancer.server.port=9001"
        
networks:
  traefik-network:
    driver: overlay
    external: true
    attachable: true

volumes:
  postgres_data:
  redis_data:
  django_media:

logs:

time="2024-03-16T20:29:25Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"portainer\":{\"entryPoints\":[\"websecure\"],\"service\":\"portainer\",\"rule\":\"Host(`portainer.domain.com`)\",\"tls\":{\"certResolver\":\"traefikresolver\"}}},\"services\":{\"portainer\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.1.237:9001\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"udp\":{}}" providerName=docker

time="2024-03-16T20:29:25Z" level=debug msg="Skipping unchanged configuration." providerName=docker

time="2024-03-16T20:29:40Z" level=debug msg="Filtering disabled container" providerName=docker container=autopmf-postgres-kqrfais771cf3soumcaao5pce

time="2024-03-16T20:29:40Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"portainer\":{\"entryPoints\":[\"websecure\"],\"service\":\"portainer\",\"rule\":\"Host(`portainer.domain.com`)\",\"tls\":{\"certResolver\":\"traefikresolver\"}}},\"services\":{\"portainer\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.1.237:9001\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"udp\":{}}" providerName=docker

time="2024-03-16T20:29:40Z" level=debug msg="Skipping unchanged configuration." providerName=docker

time="2024-03-16T20:29:55Z" level=debug msg="Filtering disabled container" providerName=docker container=autopmf-postgres-kqrfais771cf3soumcaao5pce

time="2024-03-16T20:29:55Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"portainer\":{\"entryPoints\":[\"websecure\"],\"service\":\"portainer\",\"rule\":\"Host(`portainer.domain.com`)\",\"tls\":{\"certResolver\":\"traefikresolver\"}}},\"services\":{\"portainer\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.1.237:9001\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"udp\":{}}" providerName=docker

time="2024-03-16T20:29:55Z" level=debug msg="Skipping unchanged configuration." providerName=docker

time="2024-03-16T20:30:10Z" level=debug msg="Filtering disabled container" providerName=docker container=autopmf-postgres-kqrfais771cf3soumcaao5pce

time="2024-03-16T20:30:10Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"portainer\":{\"entryPoints\":[\"websecure\"],\"service\":\"portainer\",\"rule\":\"Host(`portainer.domain.com`)\",\"tls\":{\"certResolver\":\"traefikresolver\"}}},\"services\":{\"portainer\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.1.237:9001\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"udp\":{}}" providerName=docker

time="2024-03-16T20:30:10Z" level=debug msg="Skipping unchanged configuration." providerName=docker

time="2024-03-16T20:30:25Z" level=debug msg="Filtering disabled container" providerName=docker container=autopmf-postgres-kqrfais771cf3soumcaao5pce

time="2024-03-16T20:30:25Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"portainer\":{\"entryPoints\":[\"websecure\"],\"service\":\"portainer\",\"rule\":\"Host(`portainer.domain.com`)\",\"tls\":{\"certResolver\":\"traefikresolver\"}}},\"services\":{\"portainer\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.1.237:9001\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"udp\":{}}" providerName=docker

time="2024-03-16T20:30:25Z" level=debug msg="Skipping unchanged configuration." providerName=docker

time="2024-03-16T20:30:40Z" level=debug msg="Filtering disabled container" providerName=docker container=autopmf-postgres-kqrfais771cf3soumcaao5pce

time="2024-03-16T20:30:40Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"portainer\":{\"entryPoints\":[\"websecure\"],\"service\":\"portainer\",\"rule\":\"Host(`portainer.domain.com`)\",\"tls\":{\"certResolver\":\"traefikresolver\"}}},\"services\":{\"portainer\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.1.237:9001\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"udp\":{}}" providerName=docker

time="2024-03-16T20:30:40Z" level=debug msg="Skipping unchanged configuration." providerName=docker

time="2024-03-16T20:30:55Z" level=debug msg="Filtering disabled container" providerName=docker container=autopmf-postgres-kqrfais771cf3soumcaao5pce

time="2024-03-16T20:30:55Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"portainer\":{\"entryPoints\":[\"websecure\"],\"service\":\"portainer\",\"rule\":\"Host(`portainer.domain.com`)\",\"tls\":{\"certResolver\":\"traefikresolver\"}}},\"services\":{\"portainer\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.1.237:9001\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"udp\":{}}" providerName=docker

time="2024-03-16T20:30:55Z" level=debug msg="Skipping unchanged configuration." providerName=docker

time="2024-03-16T20:30:58Z" level=info msg="I have to go..."

time="2024-03-16T20:30:58Z" level=info msg="Stopping server gracefully"

time="2024-03-16T20:30:58Z" level=debug msg="Waiting 10s seconds before killing connections." entryPointName=websecure

time="2024-03-16T20:30:58Z" level=error msg="accept tcp [::]:443: use of closed network connection" entryPointName=websecure

time="2024-03-16T20:30:58Z" level=error msg="Error while starting server: accept tcp [::]:443: use of closed network connection" entryPointName=websecure

time="2024-03-16T20:30:58Z" level=debug msg="Entry point websecure closed" entryPointName=websecure

time="2024-03-16T20:30:58Z" level=debug msg="Waiting 10s seconds before killing connections." entryPointName=web

time="2024-03-16T20:30:58Z" level=error msg="accept tcp [::]:80: use of closed network connection" entryPointName=web

time="2024-03-16T20:30:58Z" level=error msg="Error while starting server: accept tcp [::]:80: use of closed network connection" entryPointName=web

time="2024-03-16T20:30:58Z" level=debug msg="Entry point web closed" entryPointName=web

time="2024-03-16T20:30:58Z" level=debug msg="Waiting 10s seconds before killing connections." entryPointName=traefik

time="2024-03-16T20:30:58Z" level=error msg="accept tcp [::]:8080: use of closed network connection" entryPointName=traefik

time="2024-03-16T20:30:58Z" level=error msg="Error while starting server: accept tcp [::]:8080: use of closed network connection" entryPointName=traefik

time="2024-03-16T20:30:58Z" level=debug msg="Entry point traefik closed" entryPointName=traefik

time="2024-03-16T20:30:58Z" level=info msg="Server stopped"

time="2024-03-16T20:30:58Z" level=info msg="Shutting down"
2

It would be great to get a better diagnose than "not working". What’s not working? Are all containers up? Did you check Traefik dashboard?

Usually you would just expose ports on Traefik, not on the other services/containers. Otherwise Traefik security middlewares can be circumvented. Traefik should forward requests internally over Docker network, in which all ports are reachable.

Portainer should work with internal port 9000, check this post from this week.

Did you use docker stack deploy? Compare to working simple Traefik Swarm example.

might be shown during shutdown.