Hi all,
First of all i would like to wish you all.. Happy New Year!!!
I'm struggling with making Traefik 2 working in swarm mode. Let me explain how it works and how it is not working:
My Setup: 5 Raspberry Pi (3 Masters, 2 Workers) + Cloudflare in the back.
When i have started with my home project i just used 1 Rpi. Traefik worked as expected.
For the last few month I have added another 4 of them, re-done the project and Traefik works only on the first node.
If I drain the first node Traefik is not working anymore, even if its active on the node 2 and node 3 (masters as well)
I will copy and paste from github the following:
What did you see instead?
Error 521 on Cloudflare
What version of Traefik are you using?
2.5.6
What is your environment & configuration?
Partial, without other containers:
version: '3.9'
########################### NETWORKS
networks:
t2_proxy:
external: true
socket_proxy:
external: true
local:
external: true
agent_network:
external: true
########################### SECRETS
secrets:
cloudflare_email:
external: true
cloudflare_api_key:
external: true
google_client_id:
external: true
google_client_secret:
external: true
oauth_secret:
external: true
my_email:
external: true
cloudflare_api_token:
external: true
mysql_root_password:
external: true
########################### SERVICES
services:
########################### Traefik 2 - Reverse Proxy
traefik2:
image: traefik:2.5 # brie v2.5.x livarot v2.4.x # picodon v2.3.x
networks:
- t2_proxy
- socket_proxy
command: # CLI arguments
- --global.checkNewVersion=true
- --global.sendAnonymousUsage=true
- --entryPoints.http.address=:80
- --entryPoints.https.address=:443
# Allow these IPs to set the X-Forwarded-* headers - Cloudflare IPs: https://www.cloudflare.com/ips/
- --entrypoints.https.forwardedHeaders.trustedIPs=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22
- --entryPoints.traefik.address=:8080
- --api=true
# - --api.insecure=true
- --api.dashboard=true
- --serversTransport.insecureSkipVerify=true
- --log=true
- --log.level=DEBUG # (Default: error) DEBUG, INFO, WARN, ERROR, FATAL, PANIC
- --accessLog=true
- --accessLog.filePath=/traefik.log
- --accessLog.bufferingSize=100 # Configuring a buffer of 100 lines
- --accessLog.filters.statusCodes=400-499
- --providers.docker=true
#- --providers.docker.endpoint=unix:///var/run/docker.sock
- --providers.docker.endpoint=tcp://socket-proxy:2375
# - --providers.docker.defaultrule=Host(`{{ index .Labels "com.docker.compose.service" }}.$DOMAINNAME`)
- --providers.docker.exposedByDefault=false
# - --entrypoints.https.http.middlewares=chain-oauth@file
# - --entrypoints.https.http.tls.options=tls-opts@file
# Add dns-cloudflare as default certresolver for all services. Also enables TLS and no need to specify on individual services
- --entrypoints.https.http.tls.certresolver=dns-cloudflare
- --entrypoints.https.http.tls.domains[0].main=$DOMAINNAME
- --entrypoints.https.http.tls.domains[0].sans=*.$DOMAINNAME
# - --entrypoints.https.http.tls.domains[1].main=$DOMAINNAME1 # Pulls main cert for second domain
# - --entrypoints.https.http.tls.domains[1].sans=*.$DOMAINNAME1 # Pulls wildcard cert for second domain
- --providers.docker.network=t2_proxy
- --providers.docker.swarmMode=true
- --providers.file.directory=/rules # Load dynamic configuration from one or more .toml or .yml files in a directory
# - --providers.file.filename=/path/to/file # Load dynamic configuration from a file
- --providers.file.watch=true # Only works on top level files in the rules folder
# - --certificatesResolvers.dns-cloudflare.acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory # LetsEncrypt Staging Server - uncomment when testing
- --certificatesResolvers.dns-cloudflare.acme.email=$CLOUDFLARE_EMAIL
- --certificatesResolvers.dns-cloudflare.acme.storage=/acme.json
- --certificatesResolvers.dns-cloudflare.acme.dnsChallenge.provider=cloudflare
- --certificatesResolvers.dns-cloudflare.acme.dnsChallenge.resolvers=1.1.1.1:53,1.0.0.1:53
- --certificatesResolvers.dns-cloudflare.acme.dnsChallenge.delayBeforeCheck=90 # To delay DNS check and reduce LE hitrate
ports:
- target: 80
published: 80
protocol: tcp
mode: host
- target: 443
published: 443
protocol: tcp
mode: host
- target: 8080
published: 8080
protocol: tcp
mode: host
volumes:
- $DOCKERDIR/traefik2/rules:/rules
# - /var/run/docker.sock:/var/run/docker.sock:ro
- $DOCKERDIR/traefik2/acme/acme.json:/acme.json
- $DOCKERDIR/traefik2/traefik.log:/traefik.log
- $DOCKERDIR/shared:/shared
environment:
- CF_API_EMAIL_FILE=/run/secrets/cloudflare_email
- CF_API_KEY_FILE=/run/secrets/cloudflare_api_key
secrets:
- cloudflare_email
- cloudflare_api_key
deploy:
mode: global
placement:
constraints:
- node.role == manager
update_config:
parallelism: 1
delay: 10s
restart_policy:
condition: any
labels:
- "traefik.enable=true"
# HTTP-to-HTTPS Redirect
- "traefik.http.routers.http-catchall.entrypoints=http"
- "traefik.http.routers.http-catchall.rule=HostRegexp(`{host:.+}`)"
- "traefik.http.routers.http-catchall.middlewares=redirect-to-https"
- "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
# HTTP Routers
- "traefik.http.routers.traefik-rtr.entrypoints=https"
- "traefik.http.routers.traefik-rtr.rule=Host(`traefik.$DOMAINNAME`)"
## Services - API
- "traefik.http.routers.traefik-rtr.service=api@internal"
## Healthcheck/ping
#- "traefik.http.routers.ping.rule=Host(`traefik.$DOMAINNAME`) && Path(`/ping`)"
#- "traefik.http.routers.ping.tls=true"
#- "traefik.http.routers.ping.service=ping@internal"
## Middlewares
- "traefik.http.services.traefik-rtr.loadbalancer.server.port=80"
- "traefik.http.routers.traefik-rtr.middlewares=chain-basic-auth@file"
- "traefik.http.routers.traefik-rtr.middlewares=chain-oauth@file"
yml executed with the following command:
docker stack deploy -c <(docker-compose -f /home/pi/mnt/traefik2.yml config) traefik2 --resolve-image=never
If applicable, please paste the log output in DEBUG level
79.113.192.25 - - [02/Jan/2022:15:52:18 +0000] "GET /api/users/1 HTTP/2.0" 401 78 "-" "-" 2 "portainer-rtr@docker" "http://10.11.0.10:9000" 13ms
79.113.192.25 - - [02/Jan/2022:15:52:20 +0000] "GET /api/endpoints?limit=1&start=0 HTTP/2.0" 401 78 "-" "-" 6 "portainer-rtr@docker" "http://10.11.0.10:9000" 8ms
79.113.192.25 - - [02/Jan/2022:15:52:20 +0000] "GET /api/status/version HTTP/2.0" 401 78 "-" "-" 8 "portainer-rtr@docker" "http://10.11.0.10:9000" 8ms
192.168.20.2 - - [02/Jan/2022:16:42:15 +0000] "GET / HTTP/1.1" - - "-" "-" 1 "-" "-" 0ms
192.168.20.2 - - [02/Jan/2022:16:42:16 +0000] "GET /favicon.ico HTTP/1.1" - - "-" "-" 2 "-" "-" 0ms
192.168.20.2 - - [02/Jan/2022:16:42:24 +0000] "GET / HTTP/2.0" 404 19 "-" "-" 4 "-" "-" 0ms
192.168.20.2 - - [02/Jan/2022:16:42:25 +0000] "GET /favicon.ico HTTP/2.0" 404 19 "-" "-" 5 "-" "-" 0ms
192.168.20.2 - - [02/Jan/2022:16:49:53 +0000] "GET / HTTP/2.0" - - "-" "-" 6 "-" "-" 0ms
192.168.20.2 - - [02/Jan/2022:16:49:53 +0000] "GET /favicon.ico HTTP/2.0" - - "-" "-" 7 "-" "-" 0ms
192.168.20.2 - - [02/Jan/2022:16:49:57 +0000] "GET / HTTP/2.0" - - "-" "-" 8 "-" "-" 0ms
192.168.20.2 - - [02/Jan/2022:17:01:00 +0000] "GET / HTTP/2.0" - - "-" "-" 1 "-" "-" 0ms
79.113.200.90 - - [02/Jan/2022:17:53:20 +0000] "GET / HTTP/2.0" 404 19 "-" "-" 1 "-" "-" 0ms
79.113.200.90 - - [02/Jan/2022:17:53:20 +0000] "GET /favicon.ico HTTP/2.0" 404 19 "-" "-" 2 "-" "-" 0ms
192.168.20.2 - - [02/Jan/2022:18:00:49 +0000] "GET / HTTP/2.0" 404 19 "-" "-" 205 "-" "-" 0ms
192.168.20.2 - - [02/Jan/2022:18:00:49 +0000] "GET /favicon.ico HTTP/2.0" 404 19 "-" "-" 206 "-" "-" 0ms
192.168.20.2 - - [02/Jan/2022:18:00:55 +0000] "GET / HTTP/2.0" - - "-" "-" 207 "-" "-" 0ms
192.168.20.2 - - [02/Jan/2022:18:00:55 +0000] "GET /favicon.ico HTTP/2.0" - - "-" "-" 208 "-" "-" 0ms
192.168.20.2 - - [02/Jan/2022:18:01:04 +0000] "GET / HTTP/1.1" - - "-" "-" 209 "-" "-" 0ms
192.168.20.2 - - [02/Jan/2022:18:01:04 +0000] "GET /favicon.ico HTTP/1.1" - - "-" "-" 210 "-" "-" 0ms
I have attached the log file from a Master node that is not working.
https://github.com/traefik/traefik/files/7799571/_traefik2_traefik2.hxiaxw6zfxyrwr6mo23hvxe9i.vxl3vlefbk9hqdsu67era2i4f_logs.txt