Gateway 504 error with services

Traefik Traefik v2
1

Hi there, I am using traefik 2.5.0 in docker swarm mode. When I deploy my stack I am able to reach the traefik frontend without issue. But when I try to reach my authelia frontend I get a gateway 504 timeout. I have made a telnet to the ports 443 and 80 from an external source and they both connect. I also inspected the internal IP addresses and they match the correct network for connection. I see nothing in the logs when I try to access this only what I have below:

LOGS from Traefik

traefik_traefik.1.idhmosli9ld7@node-master    | time="2022-05-11T06:28:23Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/favicon.ico\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip\"],\"Accept-Language\":[\"en-GB,en;q=0.9\"],\"Cdn-Loop\":[\"cloudflare\"],\"Cf-Connecting-Ip\":[\"123.123.123.123\"],\"Cf-Ipcountry\":[\"CZ\"],\"Cf-Ray\":[\"7098dab02a626b2a-AMS\"],\"Cf-Visitor\":[\"{\\\"scheme\\\":\\\"https\\\"}\"],\"Cookie\":[\"cf_ob_info=504:7098d9ae3df56b2a:AMS; cf_use_ob=0\"],\"Referer\":[\"https://authelia.domain.tld/\"],\"Sec-Ch-Ua\":[\"\\\" Not A;Brand\\\";v=\\\"99\\\", \\\"Chromium\\\";v=\\\"101\\\", \\\"Google Chrome\\\";v=\\\"101\\\"\"],\"Sec-Ch-Ua-Mobile\":[\"?0\"],\"Sec-Ch-Ua-Platform\":[\"\\\"Windows\\\"\"],\"Sec-Fetch-Dest\":[\"image\"],\"Sec-Fetch-Mode\":[\"no-cors\"],\"Sec-Fetch-Site\":[\"same-origin\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36\"],\"X-Forwarded-For\":[\"123.123.123.123\"],\"X-Forwarded-Host\":[\"authelia.domain.tld\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"be3b9a89b041\"],\"X-Real-Ip\":[\"141.101.104.87\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"authelia.domain.tld\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"141.101.104.87:15786\",\"RequestURI\":\"/favicon.ico\",\"TLS\":null}"
    traefik_traefik.1.idhmosli9ld7@node-master    | time="2022-05-11T06:28:23Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" ForwardURL="http://10.0.17.3:9091" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/favicon.ico\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip\"],\"Accept-Language\":[\"en-GB,en;q=0.9\"],\"Cdn-Loop\":[\"cloudflare\"],\"Cf-Connecting-Ip\":[\"123.123.123.123\"],\"Cf-Ipcountry\":[\"CZ\"],\"Cf-Ray\":[\"7098dab02a626b2a-AMS\"],\"Cf-Visitor\":[\"{\\\"scheme\\\":\\\"https\\\"}\"],\"Cookie\":[\"cf_ob_info=504:7098d9ae3df56b2a:AMS; cf_use_ob=0\"],\"Referer\":[\"https://authelia.domain.tld/\"],\"Sec-Ch-Ua\":[\"\\\" Not A;Brand\\\";v=\\\"99\\\", \\\"Chromium\\\";v=\\\"101\\\", \\\"Google Chrome\\\";v=\\\"101\\\"\"],\"Sec-Ch-Ua-Mobile\":[\"?0\"],\"Sec-Ch-Ua-Platform\":[\"\\\"Windows\\\"\"],\"Sec-Fetch-Dest\":[\"image\"],\"Sec-Fetch-Mode\":[\"no-cors\"],\"Sec-Fetch-Site\":[\"same-origin\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36\"],\"X-Forwarded-For\":[\"123.123.123.123\"],\"X-Forwarded-Host\":[\"authelia.domain.tld\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"be3b9a89b041\"],\"X-Real-Ip\":[\"141.101.104.87\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"authelia.domain.tld\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"141.101.104.87:15786\",\"RequestURI\":\"/favicon.ico\",\"TLS\":null}"


traefik_traefik.1.idhmosli9ld7@node-master    | time="2022-05-11T06:27:12Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip\"],\"Accept-Language\":[\"en-GB,en;q=0.9\"],\"Cache-Control\":[\"max-age=0\"],\"Cdn-Loop\":[\"cloudflare\"],\"Cf-Connecting-Ip\":[\"123.123.123.123\"],\"Cf-Ipcountry\":[\"CZ\"],\"Cf-Ray\":[\"7098d8f09d0a6b2a-AMS\"],\"Cf-Visitor\":[\"{\\\"scheme\\\":\\\"https\\\"}\"],\"Referer\":[\"https://authelia.domain.tld/\"],\"Sec-Ch-Ua\":[\"\\\" Not A;Brand\\\";v=\\\"99\\\", \\\"Chromium\\\";v=\\\"101\\\", \\\"Google Chrome\\\";v=\\\"101\\\"\"],\"Sec-Ch-Ua-Mobile\":[\"?0\"],\"Sec-Ch-Ua-Platform\":[\"\\\"Windows\\\"\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Fetch-Site\":[\"same-origin\"],\"Sec-Fetch-User\":[\"?1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 
10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36\"],\"X-Forwarded-For\":[\"123.123.123.123\"],\"X-Forwarded-Host\":[\"authelia.domain.tld\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"be3b9a89b041\"],\"X-Real-Ip\":[\"141.101.76.110\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"authelia.domain.tld\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"141.101.76.110:33848\",\"RequestURI\":\"/\",\"TLS\":null}"
traefik_traefik.1.idhmosli9ld7@node-master    | time="2022-05-11T06:27:12Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip\"],\"Accept-Language\":[\"en-GB,en;q=0.9\"],\"Cache-Control\":[\"max-age=0\"],\"Cdn-Loop\":[\"cloudflare\"],\"Cf-Connecting-Ip\":[\"123.123.123.123\"],\"Cf-Ipcountry\":[\"CZ\"],\"Cf-Ray\":[\"7098d8f09d0a6b2a-AMS\"],\"Cf-Visitor\":[\"{\\\"scheme\\\":\\\"https\\\"}\"],\"Referer\":[\"https://authelia.domain.tld/\"],\"Sec-Ch-Ua\":[\"\\\" Not A;Brand\\\";v=\\\"99\\\", \\\"Chromium\\\";v=\\\"101\\\", \\\"Google Chrome\\\";v=\\\"101\\\"\"],\"Sec-Ch-Ua-Mobile\":[\"?0\"],\"Sec-Ch-Ua-Platform\":[\"\\\"Windows\\\"\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Fetch-Site\":[\"same-origin\"],\"Sec-Fetch-User\":[\"?1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36\"],\"X-Forwarded-For\":[\"123.123.123.123\"],\"X-Forwarded-Host\":[\"authelia.domain.tld\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"be3b9a89b041\"],\"X-Real-Ip\":[\"141.101.76.110\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"authelia.domain.tld\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"141.101.76.110:33848\",\"RequestURI\":\"/\",\"TLS\":null}" ForwardURL="http://10.0.17.3:9091"
traefik_traefik.1.idhmosli9ld7@node-master    | time="2022-05-11T06:27:19Z" level=debug msg="Configuration received from provider docker: {\"http\":{\"routers\":{\"authelia\":{\"entryPoints\":[\"https\"],\"service\":\"authelia\",\"rule\":\"Host(`authelia.domain.tld`)\",\"tls\":{}},\"http-catchall\":{\"entryPoints\":[\"http\"],\"middlewares\":[\"redirect-to-https\"],\"service\":\"traefik\",\"rule\":\"HostRegexp(`{host:.+}`)\"},\"portainer\":{\"middlewares\":[\"authelia@docker\"],\"service\":\"portainer-svc\",\"rule\":\"Host(`traefik-portainer`)\"},\"portainer-rtr\":{\"entryPoints\":[\"https\"],\"middlewares\":[\"chain-authelia@file\"],\"service\":\"portainer-svc\",\"rule\":\"Host(`portainer.domain.tld`)\",\"tls\":{}},\"traefik-rtr\":{\"entryPoints\":[\"https\"],\"middlewares\":[\"chain-basic-auth@file\"],\"service\":\"api@internal\",\"rule\":\"Host(`monitor.domain.tld`)\",\"tls\":{}}},\"services\":{\"authelia\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.17.3:9091\"}],\"passHostHeader\":true}},\"portainer-svc\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.3.73:9000\"}],\"passHostHeader\":true}},\"traefik\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.3.67:80\"}],\"passHostHeader\":true}}},\"middlewares\":{\"authelia\":{\"forwardAuth\":{\"address\":\"http://authelia:9091/api/verify?rd=https://authelia.domain.tld/\",\"trustForwardHeader\":true,\"authResponseHeaders\":[\"Remote-User\",\"Remote-Groups\"]}},\"redirect-to-https\":{\"redirectScheme\":{\"scheme\":\"https\"}}}},\"tcp\":{},\"udp\":{}}" providerName=docker
traefik_traefik.1.idhmosli9ld7@node-master    | time="2022-05-11T06:27:19Z" level=debug msg="Creating middleware" middlewareType=ForwardedAuthType entryPointName=http routerName=portainer@docker middlewareName=authelia@docker
traefik_traefik.1.idhmosli9ld7@node-master    | time="2022-05-11T06:27:19Z" level=debug msg="Creating middleware" entryPointName=https routerName=portainer-rtr@docker middlewareName=chain-authelia@file middlewareType=Chain
traefik_traefik.1.idhmosli9ld7@node-master    | time="2022-05-11T06:27:19Z" level=debug msg="Creating middleware" entryPointName=https routerName=portainer-rtr@docker middlewareName=middlewares-authelia@file middlewareType=ForwardedAuthType
traefik_traefik.1.idhmosli9ld7@node-master    | time="2022-05-11T06:27:19Z" level=debug msg="Creating middleware" entryPointName=https routerName=authelia@docker serviceName=authelia middlewareName=pipelining middlewareType=Pipelining
traefik_traefik.1.idhmosli9ld7@node-master    | time="2022-05-11T06:27:19Z" level=debug msg="Creating load-balancer" entryPointName=https routerName=authelia@docker serviceName=authelia
traefik_traefik.1.idhmosli9ld7@node-master    | time="2022-05-11T06:27:19Z" level=debug msg="Creating server 0 http://10.0.17.3:9091" entryPointName=https routerName=authelia@docker serviceName=authelia serverName=0
traefik_traefik.1.idhmosli9ld7@node-master    | time="2022-05-11T06:27:19Z" level=debug msg="Added outgoing tracing middleware authelia" middlewareName=tracing middlewareType=TracingForwarder routerName=authelia@docker entryPointName=https
traefik_traefik.1.idhmosli9ld7@node-master    | time="2022-05-11T06:27:19Z" level=debug msg="Adding route for authelia.domain.tld with TLS options default" entryPointName=https

Here is my compose file:

version: "3.7"

networks:
  t2_proxy:
    external:
      name: t2_proxy
  dbnet:
   driver: overlay
   name: dbnet

secrets:
  authelia_jwt_secret:
    file: ./secrets/authelia/authelia_jwt_secret
  authelia_session_secret:
    file: ./secrets/authelia/authelia_session_secret
  authelia_notifier_smtp_password:
    file: ./secrets/authelia/authelia_notifier_smtp_password
  cloudflare_email:
    file: ./secrets/traefik/cloudflare_email
  cloudflare_api_key:
    file: ./secrets/traefik/cloudflare_api_key
  authelia_db_name:
    file: ./secrets/mysql/authelia_db_name
  authelia_db_user:
    file: ./secrets/mysql/authelia_db_user
  authelia_db_password:
    file: ./secrets/mysql/authelia_db_password
  mysql_root_password:
    file: ./secrets/mysql/mysql_root_password

services:
  # Traefik 2 - Reverse Proxy
  traefik:
    image: traefik:2.5.0
    deploy:
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.traefik-rtr.rule=Host(`monitor.domain.tld`)"
        - "traefik.http.routers.http-catchall.entrypoints=http"
        - "traefik.http.routers.traefik-rtr.entrypoints=https"
        - 'traefik.http.services.traefik.loadbalancer.server.port=80'
        - "traefik.http.routers.http-catchall.rule=HostRegexp(`{host:.+}`)"
        - "traefik.http.routers.http-catchall.middlewares=redirect-to-https"
        - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
        - "traefik.http.routers.traefik-rtr.tls=true"
        - "traefik.http.routers.traefik-rtr.service=api@internal"
        - "traefik.http.routers.traefik-rtr.middlewares=chain-authelia@file"
        - "traefik.http.routers.traefik-rtr.middlewares=chain-basic-auth@file"
    command:
      - --entryPoints.http.address=:80
      - --entryPoints.https.address=:443
      - --entrypoints.https.forwardedHeaders.trustedIPs=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/12,172.64.0.0/13,131.0.72.0/22
      - --entryPoints.traefik.address=:8080
      - --api=true
      - --log=true
      - --log.level=DEBUG # (Default: error) DEBUG, INFO, WARN, ERROR, FATAL, PANIC
      - --accessLog=true
      - --accessLog.filePath=/traefik.log
      - --accessLog.bufferingSize=100 # Configuring a buffer of 100 lines
      - --accessLog.filters.statusCodes=400-499
      - --providers.docker=true
      - --providers.docker.exposedByDefault=false
      - --providers.docker.swarmMode=true
      - --providers.file.directory=/rules
      - --providers.file.watch=true # Only works on top level files in the rules folder
      - --certificatesResolvers.dns-cloudflare.acme.email=$CLOUDFLARE_EMAIL
      - --certificatesResolvers.dns-cloudflare.acme.storage=/acme.json
      - --certificatesResolvers.dns-cloudflare.acme.dnsChallenge.provider=cloudflare
      - --certificatesResolvers.dns-cloudflare.acme.dnsChallenge.resolvers=1.1.1.1:53,1.0.0.1:53
      - --certificatesResolvers.dns-cloudflare.acme.dnsChallenge.delayBeforeCheck=90
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
    volumes:
      - ./traefik2/rules:/rules
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./traefik2/acme/acme.json:/acme.json
      - ./traefik2/traefik.log:/traefik.log
      - ./shared:/shared
    environment:
      - CF_API_EMAIL=/run/secrets/cloudflare_email
      - CF_API_KEY=/run/secrets/cloudflare_api_key
    secrets:
      - cloudflare_email
      - cloudflare_api_key
    networks:
      - t2_proxy

  redis:
      image: redis:6-alpine
      volumes:
        - ./redis:/data
      networks:
        - t2_proxy

  authelia:
    image: authelia/authelia:latest
    depends_on:
      - mariadb
    restart: always
    networks:
      - t2_proxy
      - dbnet
    volumes:
      - ./authelia:/config
  #    - ./secrets:/config/secrets
    environment:
      - TZ=Europe/Prague
      - AUTHELIA_JWT_SECRET_FILE=/run/secrets/authelia_jwt_secret
      - AUTHELIA_SESSION_SECRET_FILE=/run/secrets/authelia_session_secret
      - AUTHELIA_STORAGE_MYSQL_PASSWORD_FILE=/run/secrets/authelia_db_password
      - AUTHELIA_NOTIFIER_SMTP_PASSWORD_FILE=/run/secrets/authelia_notifier_smtp_password
    secrets:
      - authelia_jwt_secret
      - authelia_session_secret
      - authelia_notifier_smtp_password
      - authelia_db_password
    deploy:
      labels:
        - 'traefik.enable=true'
        - 'traefik.http.routers.authelia.rule=Host(`auth.domain.tld`)'
        - 'traefik.http.routers.authelia.entrypoints=https'
        - "traefik.http.services.authelia.loadbalancer.server.port=9091"
        - 'traefik.http.routers.authelia.tls=true'
        - 'traefik.http.middlewares.authelia.forwardauth.address=http://authelia:9091/api/verify?rd=https://auth.domain.tld/'
        - 'traefik.http.middlewares.authelia.forwardauth.trustForwardHeader=true'
        - 'traefik.http.middlewares.authelia.forwardauth.authResponseHeaders=Remote-User, Remote-Groups'

  mariadb:
    image: linuxserver/mariadb:latest
    restart: always
    networks:
      - dbnet
    environment:
      MYSQL_DATABASE: /run/secrets/authelia_db_name
      MYSQL_USER: /run/secrets/auhelia_db_user
      MYSQL_PASSWORD: /run/secrets/authelia_db_password
      MYSQL_ROOT_PASSWORD: /run/secrets/mysql_root_password
    secrets:
      - authelia_db_name
      - authelia_db_user
      - authelia_db_password
      - mysql_root_password
    ports:
      - "3306:3306"
    volumes:
      - ./mariadb/data:/config
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

  portainer:
    container_name: portainer
    image: portainer/portainer-ce:2.11.1
    restart: unless-stopped
    command: -H unix:///var/run/docker.sock
    depends_on:
      - authelia
      - traefik
    networks:
      - t2_proxy
      - dbnet
    security_opt:
      - no-new-privileges:true
#    ports:
#      - "$PORTAINER_PORT:9000"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./portainer/data:/data
    environment:
      - TZ=Europe/Prague
    deploy:
      labels:
        - "traefik.enable=true"
        ## HTTP Routers
        - "traefik.http.routers.portainer-rtr.entrypoints=https"
        - "traefik.http.routers.portainer-rtr.rule=Host(`portainer.domain.tld`)"
        - "traefik.http.routers.portainer-rtr.tls=true"
        ## Middlewares
        - "traefik.http.routers.portainer-rtr.middlewares=chain-no-auth@file" # No Authentication
        ## HTTP Services
        - "traefik.http.routers.portainer-rtr.service=portainer-svc"
        - "traefik.http.services.portainer-svc.loadbalancer.server.port=9000"
        - "traefik.http.routers.portainer-rtr.middlewares=chain-authelia@file"
        - 'traefik.http.routers.portainer.middlewares=authelia@docker'

configuration.yml

###############################################################
#                   Authelia configuration                    #
###############################################################

server.host: 0.0.0.0
server.port: 9091
log.level: debug

jwt_secret:
default_redirection_url: https://authelia.domain.tld

totp:
  issuer: authelia.com
  period: 30
  skew: 1

authentication_backend:
  file:
    path: /config/users_database.yml
    password:
      algorithm: argon2id
      iterations: 1
      salt_length: 16
      parallelism: 8
      memory: 1024 # blocks this much of the RAM. Tune this.

access_control:
  default_policy: deny
  rules:
    - domain: "*"
      policy: bypass
      networks:
        - 192.168.1.0/24
    - domain:
        - "*.domain.tld"
        - "domain.tld"
      policy: two_factor

session:
  name: authelia_session
  expiration: 3600 # 1 hour
  inactivity: 1200 # 5 minutes
  domain: domain.tld # Should match whatever your root protected domain is

regulation:
  max_retries: 3
  find_time: 120
  ban_time: 300

storage:
  encryption_key: Ffs*rN@Wc,~P^p"rY^uVfF.r[K<?$Somencryption
  mysql:
    host: mariadb
    port: 3306
    database: authelia
    username: authelia

notifier:
  disable_startup_check: false
  smtp:
    timeout: 5s
    username: mail@domain.de
    host: smtp.zoho.eu
    port: 587
    identifier: localhost
    sender: mail@domain.de
    subject: "[Authelia] {title}"
    startup_check_address: test@authelia.com
    disable_require_tls: false
    disable_html_emails: false
    tls:
      server_name: smtp.zoho.eu
      skip_verify: false
      minimum_version: TLS1.2

middleware.chains.toml

[http.middlewares]
  [http.middlewares.chain-no-auth]
    [http.middlewares.chain-no-auth.chain]
      middlewares = [ "middlewares-rate-limit", "middlewares-https-redirectscheme", "middlewares-secure-headers"]

  [http.middlewares.chain-basic-auth]
    [http.middlewares.chain-basic-auth.chain]
      middlewares = [ "middlewares-rate-limit", "middlewares-https-redirectscheme", "middlewares-secure-headers", "middlewares-basic-auth"]

  [http.middlewares.chain-authelia]
    [http.middlewares.chain-authelia.chain]
      middlewares = [ "middlewares-rate-limit", "middlewares-https-redirectscheme", "middlewares-secure-headers", "middlewares-authelia"]

  [http.middlewares.chain-nextcloud]
    [http.middlewares.chain-nextcloud.chain]
      middlewares = [ "middlewares-rate-limit", "middlewares-https-redirectscheme", "nextcloud-middlewares-secure-headers", "nextcloud-redirect"]

middlewares.toml

[http.middlewares]
  [http.middlewares.middlewares-basic-auth]
    [http.middlewares.middlewares-basic-auth.basicAuth]
      realm = "Traefik2 Basic Auth"
      usersFile = "/shared/.htpasswd" #be sure to mount the volume through docker-compose.yml

  [http.middlewares.middlewares-rate-limit]
    [http.middlewares.middlewares-rate-limit.rateLimit]
      average = 100
      burst = 50

  [http.middlewares.middlewares-secure-headers]
    [http.middlewares.middlewares-secure-headers.headers]
      accessControlAllowMethods= ["GET", "OPTIONS", "PUT"]
      accessControlMaxAge = 100
      hostsProxyHeaders = ["X-Forwarded-Host"]
      stsSeconds = 63072000
      stsIncludeSubdomains = true
      stsPreload = true
      forceSTSHeader = true
      customFrameOptionsValue = "allow-from https:domain.tld" #CSP takes care of this but may be needed for organizr.
      contentTypeNosniff = true
      browserXssFilter = true
      referrerPolicy = "same-origin"
      permissionsPolicy = "camera=(), microphone=(), geolocation=(), payment=(), usb=(), vr=()"
      [http.middlewares.middlewares-secure-headers.headers.customResponseHeaders]
        X-Robots-Tag = "none,noarchive,nosnippet,notranslate,noimageindex,"
        server = ""

  [http.middlewares.middlewares-authelia]
    [http.middlewares.middlewares-authelia.forwardAuth]
      address = "http://authelia:9091/api/verify?rd=https://authelia.domain.tld"
      trustForwardHeader = true
      authResponseHeaders = ["Remote-User", "Remote-Groups"]


  [http.middlewares.middlewares-https-redirectscheme]
    [http.middlewares.middlewares-https-redirectscheme.redirectScheme]
      scheme = "https"
      permanent = true


  [http.middlewares.nextcloud-redirectregex.redirectRegex]
    permanent = true
    regex = "https://(.*)/.well-known/(card|cal)dav"
    replacement = "https://${1}/remote.php/dav/"
2

Hi @dynamicat,
Thanks for your interest in Traefik.

I'm trying to reproduce your issue, but I would need more information on your setup.
Could you provide your Traefik Proxy configuration (e.g: docker-compose conf)?

3

Hi there sorry that was my bad.
@moutoum

version: "3.7"

networks:
  t2_proxy:
    external:
      name: t2_proxy
  dbnet:
   driver: overlay
   name: dbnet

secrets:
  authelia_jwt_secret:
    file: ./secrets/authelia/authelia_jwt_secret
  authelia_session_secret:
    file: ./secrets/authelia/authelia_session_secret
  authelia_notifier_smtp_password:
    file: ./secrets/authelia/authelia_notifier_smtp_password
  cloudflare_email:
    file: ./secrets/traefik/cloudflare_email
  cloudflare_api_key:
    file: ./secrets/traefik/cloudflare_api_key
  authelia_db_name:
    file: ./secrets/mysql/authelia_db_name
  authelia_db_user:
    file: ./secrets/mysql/authelia_db_user
  authelia_db_password:
    file: ./secrets/mysql/authelia_db_password
  mysql_root_password:
    file: ./secrets/mysql/mysql_root_password

services:
  # Traefik 2 - Reverse Proxy
  traefik:
    image: traefik:2.5.0
    deploy:
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.traefik-rtr.rule=Host(`monitor.domain.tld`)"
        - "traefik.http.routers.http-catchall.entrypoints=http"
        - "traefik.http.routers.traefik-rtr.entrypoints=https"
        - 'traefik.http.services.traefik.loadbalancer.server.port=80'
        - "traefik.http.routers.http-catchall.rule=HostRegexp(`{host:.+}`)"
        - "traefik.http.routers.http-catchall.middlewares=redirect-to-https"
        - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
        - "traefik.http.routers.traefik-rtr.tls=true"
        - "traefik.http.routers.traefik-rtr.service=api@internal"
        - "traefik.http.routers.traefik-rtr.middlewares=chain-authelia@file"
        - "traefik.http.routers.traefik-rtr.middlewares=chain-basic-auth@file"
    command:
      - --entryPoints.http.address=:80
      - --entryPoints.https.address=:443
      - --entrypoints.https.forwardedHeaders.trustedIPs=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/12,172.64.0.0/13,131.0.72.0/22
      - --entryPoints.traefik.address=:8080
      - --api=true
      - --log=true
      - --log.level=DEBUG # (Default: error) DEBUG, INFO, WARN, ERROR, FATAL, PANIC
      - --accessLog=true
      - --accessLog.filePath=/traefik.log
      - --accessLog.bufferingSize=100 # Configuring a buffer of 100 lines
      - --accessLog.filters.statusCodes=400-499
      - --providers.docker=true
      - --providers.docker.exposedByDefault=false
      - --providers.docker.swarmMode=true
      - --providers.file.directory=/rules
      - --providers.file.watch=true # Only works on top level files in the rules folder
      - --certificatesResolvers.dns-cloudflare.acme.email=$CLOUDFLARE_EMAIL
      - --certificatesResolvers.dns-cloudflare.acme.storage=/acme.json
      - --certificatesResolvers.dns-cloudflare.acme.dnsChallenge.provider=cloudflare
      - --certificatesResolvers.dns-cloudflare.acme.dnsChallenge.resolvers=1.1.1.1:53,1.0.0.1:53
      - --certificatesResolvers.dns-cloudflare.acme.dnsChallenge.delayBeforeCheck=90
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
    volumes:
      - ./traefik2/rules:/rules
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./traefik2/acme/acme.json:/acme.json
      - ./traefik2/traefik.log:/traefik.log
      - ./shared:/shared
    environment:
      - CF_API_EMAIL=/run/secrets/cloudflare_email
      - CF_API_KEY=/run/secrets/cloudflare_api_key
    secrets:
      - cloudflare_email
      - cloudflare_api_key
    networks:
      - t2_proxy

  redis:
      image: redis:6-alpine
      volumes:
        - ./redis:/data
      networks:
        - t2_proxy

  authelia:
    image: authelia/authelia:latest
    depends_on:
      - mariadb
    restart: always
    networks:
      - t2_proxy
      - dbnet
    volumes:
      - ./authelia:/config
  #    - ./secrets:/config/secrets
    environment:
      - TZ=Europe/Prague
      - AUTHELIA_JWT_SECRET_FILE=/run/secrets/authelia_jwt_secret
      - AUTHELIA_SESSION_SECRET_FILE=/run/secrets/authelia_session_secret
      - AUTHELIA_STORAGE_MYSQL_PASSWORD_FILE=/run/secrets/authelia_db_password
      - AUTHELIA_NOTIFIER_SMTP_PASSWORD_FILE=/run/secrets/authelia_notifier_smtp_password
    secrets:
      - authelia_jwt_secret
      - authelia_session_secret
      - authelia_notifier_smtp_password
      - authelia_db_password
    deploy:
      labels:
        - 'traefik.enable=true'
        - 'traefik.http.routers.authelia.rule=Host(`auth.domain.tld`)'
        - 'traefik.http.routers.authelia.entrypoints=https'
        - "traefik.http.services.authelia.loadbalancer.server.port=9091"
        - 'traefik.http.routers.authelia.tls=true'
        - 'traefik.http.middlewares.authelia.forwardauth.address=http://authelia:9091/api/verify?rd=https://auth.domain.tld/'
        - 'traefik.http.middlewares.authelia.forwardauth.trustForwardHeader=true'
        - 'traefik.http.middlewares.authelia.forwardauth.authResponseHeaders=Remote-User, Remote-Groups'

  mariadb:
    image: linuxserver/mariadb:latest
    restart: always
    networks:
      - dbnet
    environment:
      MYSQL_DATABASE: /run/secrets/authelia_db_name
      MYSQL_USER: /run/secrets/auhelia_db_user
      MYSQL_PASSWORD: /run/secrets/authelia_db_password
      MYSQL_ROOT_PASSWORD: /run/secrets/mysql_root_password
    secrets:
      - authelia_db_name
      - authelia_db_user
      - authelia_db_password
      - mysql_root_password
    ports:
      - "3306:3306"
    volumes:
      - ./mariadb/data:/config
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

  portainer:
    container_name: portainer
    image: portainer/portainer-ce:2.11.1
    restart: unless-stopped
    command: -H unix:///var/run/docker.sock
    depends_on:
      - authelia
      - traefik
    networks:
      - t2_proxy
      - dbnet
    security_opt:
      - no-new-privileges:true
#    ports:
#      - "$PORTAINER_PORT:9000"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./portainer/data:/data
    environment:
      - TZ=Europe/Prague
    deploy:
      labels:
        - "traefik.enable=true"
        ## HTTP Routers
        - "traefik.http.routers.portainer-rtr.entrypoints=https"
        - "traefik.http.routers.portainer-rtr.rule=Host(`portainer.domain.tld`)"
        - "traefik.http.routers.portainer-rtr.tls=true"
        ## Middlewares
        - "traefik.http.routers.portainer-rtr.middlewares=chain-no-auth@file" # No Authentication
        ## HTTP Services
        - "traefik.http.routers.portainer-rtr.service=portainer-svc"
        - "traefik.http.services.portainer-svc.loadbalancer.server.port=9000"
        - "traefik.http.routers.portainer-rtr.middlewares=chain-authelia@file"
        - 'traefik.http.routers.portainer.middlewares=authelia@docker'
4

Does anyone have any ideas?

5

What is a 504 Gateway Timeout Error? A 504 Gateway Timeout Error means your web server didn't receive a timely response from another server upstream when it attempted to load one of your web pages . Put simply, your web servers aren't communicating with each other fast enough.

6

I know what a 504 is I wanted some help with the config, in case I missed something.

7

Hi @dynamicat,
I'm still trying to make it run, but on my side, I'm not able to start authelia properly.
The container is "Running" and healthy from what the status says, but for some reasons I'm not able to communicate with it. Also, I don't have any logs on the authelia container.
Just to be sure:

  • Do you have some logs in the authelia container?
  • Is Traefik able to find the authelia router (in the traefik web app)?
8

I have no logs either. Sometimes the authelia app runs and sometimes it doesn't

9

After a while, I managed to make it working with a simpler configuration. I started a traefik instance along with authelia and secured the traefik dashboard with a user.

The issue I encountered was because of my docker engine which was "locking" the authelia configuration file for some reasons. After updating and restarting my engine, it was working, or at least has some logs. Could you try to run the below command to see if it works?

docker exec <authelia_container_name> ls -l /config

Before the update and restart, this command was running indefinitely.
If it works, could you check if you have some logs in authelia? There might be some config errors described.

10

you separated the authelia configuration?

Did you actually get it working ? so that you could use authelia?

11

Hi @dynamicat,
I managed to reproduce the initial error (504 Gateway Timeout) and I found a fix for this.
The error comes from your networks, you created 2 networks and the authelia container is assigned to both of them. Traefik, while forwarding, doesn't know which network to use. So you have to specify it in your docker provider configuration:

- --providers.docker.network=t2_proxy

And as I said, other configurations seem good to me. Could you try to fix?
Let me know if you need more help :slight_smile:

3 Likes
12

Your solution worked perfect. Thank you for your help with this!

13

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.