Wildcard certs not working (using Route53)

Traefik Traefik v2
1

I am currently trying to set up wildcard certs and am having trouble. I am unsure how to debug this as there are no errors coming from Traefik. I'll first describe why I think it isn't working then post my config.

My acme.json contains the values below. Note there is no value for *.

{
"Certificates": [
      {
        "domain": {
          "main": "staging.c----------book.com"
        },
        ...
      },
      {
        "domain": {
          "main": "c----------book.com"
        },
      }
]
}

The certificate I see in safari does not contain an entry for the wildcard, only the main domain. I have checked with a friend who has wildcard certs working and he does have both the acme cert in the json file as well as being picked up by Safari.

Screenshot 2023-07-02 at 10.31.58

I am running on docker image traefik:v2.10.1 within docker swarm.

Here is my docker compose.

  traefik:
    image: "traefik:v2.10.1"
    networks:
      - traefik
    deploy:
      mode: "global"
    command:
      - "--providers.docker=true"
      - "--providers.docker.network=traefik"
      - "--providers.docker.exposedbydefault=false"
      - "--api.insecure=false"
      - "--accesslog=false"
      # Setup LetsEncrypt
      - "--certificatesresolvers.letsencrypt.acme.email=---@gmail.com"
      - "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json"
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge=true"
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge.provider=route53"
      # Set up an insecure listener that redirects all traffic to TLS
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--entrypoints.websecure.address=:443"
      # Set up the TLS configuration for our websecure listener
      - "--entrypoints.websecure.http.tls=true"
      - "--entrypoints.websecure.http.tls.certResolver=letsencrypt"
      - "--entrypoints.websecure.http.tls.domains[0].main=c----------book.com"
      - "--entrypoints.websecure.http.tls.domains[0].sans=*.c----------book.com""
    labels:
      traefik.enable: "true"
      # Global redirection: http to https
      traefik.http.routers.http-catchall.rule: HostRegexp(`{host:(www\.)?.+}`)
      traefik.http.routers.http-catchall.entrypoints: web
      traefik.http.routers.http-catchall.middlewares: wwwtohttps
      # Global redirection: https (www.) to https
      traefik.http.routers.wwwsecure-catchall.rule: HostRegexp(`{host:(www\.).+}`)
      traefik.http.routers.wwwsecure-catchall.entrypoints: websecure
      traefik.http.routers.wwwsecure-catchall.tls: "true"
      traefik.http.routers.wwwsecure-catchall.middlewares: wwwtohttps
      # middleware: http(s)://(www.) to  https://
      traefik.http.middlewares.wwwtohttps.redirectregex.regex: ^https?://(?:www\.)?(.+)
      traefik.http.middlewares.wwwtohttps.redirectregex.replacement: https://$${1}
      traefik.http.middlewares.wwwtohttps.redirectregex.permanent: "true"
    environment:
      - AWS_CONFIG_FILE=/run/secrets/traefik_aws_credentials_file
      - AWS_SHARED_CREDENTIALS_FILE=/run/secrets/traefik_aws_credentials_file
      - AWS_REGION=us-west-2
      - AWS_HOSTED_ZONE_ID=---
    secrets:
      - "traefik_aws_credentials_file"
    ports:
      - target: 443
        published: 443
        protocol: tcp
        mode: host
      - target: 80
        published: 80
        protocol: tcp
        mode: host
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "letsencrypt:/letsencrypt"

And my services use it like so

    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.svelte.rule=Host(`c----------book.com`)"
      - "traefik.http.routers.svelte.entrypoints=websecure"
      - "traefik.http.routers.svelte.tls.certresolver=letsencrypt"
      - "traefik.http.services.svelte.loadbalancer.server.port=3000"

And here are my dns entries

Screenshot 2023-07-02 at 10.43.05
Screenshot 2023-07-02 at 10.43.05844×406 16 KB

And the logs from traefik. I haven't added any logging flags but I would really expect any errors that might occur while failing to setup wildcard domains to be thrown without having to add any flags...

$ docker service logs proxy_traefik
proxy_traefik.0.rdfk5d0brf2f@cn-eu-central-2    | time="2023-06-19T17:44:49Z" level=info msg="Configuration loaded from flags."

Any help would be appreciated! My services are crashing sporadically and I think it may have to do with tls issues.

2

Show you full Traefik static and dynamic config, and docker-compose.yml.

Enable Traefik debug log.

Are you using Docker Swarm?

Is you DNS provider supporting/resolving * entries correctly to an IP address?

3

@bluepuma77 Sorry for the late reply, was on a trip.

I am using docker swarm.

I do not provide a static config, but the debug logs seem to show some static configuration being loaded. Config is being loaded via flags.

Full proxy docker compose 
version: "3.9"
services:
  traefik:
    image: "traefik:v2.10.1"
    networks:
      - traefik
    deploy:
      mode: "global"
    command:
      - "--providers.docker=true"
      - "--log.level=DEBUG"
      - "--providers.docker.network=traefik"
      - "--providers.docker.exposedbydefault=false"
      - "--api.insecure=false"
      - "--accesslog=false"
      # Setup LetsEncrypt
      - "--certificatesresolvers.letsencrypt.acme.email=---@gmail.com"
      - "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json"
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge=true"
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge.provider=route53"
      # Set up an insecure listener that redirects all traffic to TLS
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--entrypoints.websecure.address=:443"
      # Set up the TLS configuration for our websecure listener
      - "--entrypoints.websecure.http.tls=true"
      - "--entrypoints.websecure.http.tls.certResolver=letsencrypt"
      - "--entrypoints.websecure.http.tls.domains[0].main=---.com"
      - "--entrypoints.websecure.http.tls.domains[0].sans=*.---.com"
    labels:
      traefik.enable: "true"
      # Global redirection: http to https
      traefik.http.routers.http-catchall.rule: HostRegexp(`{host:(www\.)?.+}`)
      traefik.http.routers.http-catchall.entrypoints: web
      traefik.http.routers.http-catchall.middlewares: wwwtohttps
      # Global redirection: https (www.) to https
      traefik.http.routers.wwwsecure-catchall.rule: HostRegexp(`{host:(www\.).+}`)
      traefik.http.routers.wwwsecure-catchall.entrypoints: websecure
      traefik.http.routers.wwwsecure-catchall.tls: "true"
      traefik.http.routers.wwwsecure-catchall.middlewares: wwwtohttps
      # middleware: http(s)://(www.) to  https://
      traefik.http.middlewares.wwwtohttps.redirectregex.regex: ^https?://(?:www\.)?(.+)
      traefik.http.middlewares.wwwtohttps.redirectregex.replacement: https://$${1}
      traefik.http.middlewares.wwwtohttps.redirectregex.permanent: "true"
    environment:
      - AWS_CONFIG_FILE=/run/secrets/traefik_aws_credentials_file
      - AWS_SHARED_CREDENTIALS_FILE=/run/secrets/traefik_aws_credentials_file
      - AWS_REGION=us-west-2
      - AWS_HOSTED_ZONE_ID=---
    secrets:
      - "traefik_aws_credentials_file"
    ports:
      - target: 443
        published: 443
        protocol: tcp
        mode: host
      - target: 80
        published: 80
        protocol: tcp
        mode: host
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "letsencrypt:/letsencrypt"

volumes:
  letsencrypt:

secrets:
 traefik_aws_credentials_file:
   external: true

networks:
  traefik:
    external: true
Full service docker compose 
version: "3.9"
services:

  svelte-prod:
    image: ---
    networks:
      - traefik
      - postgres
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.svelte.rule=Host(`---.com`)"
      - "traefik.http.routers.svelte.entrypoints=websecure"
      - "traefik.http.routers.svelte.tls.certresolver=letsencrypt"
      - "traefik.http.services.svelte.loadbalancer.server.port=3000"
    secrets:
      - source: prod_db_url
        target: db_url
        uid: '103'
        gid: '103'
        mode: 0440
      - prod_aws_access_key_id
      - prod_aws_secret_access_key
    entrypoint: [ '/bin/sh', '-c', 'export DATABASE_URL=$$(cat /run/secrets/db_url) ; export AWS_ACCESS_KEY_ID=$$(cat /run/secrets/prod_aws_access_key_id) ; export AWS_SECRET_ACCESS_KEY=$$(cat /run/secrets/prod_aws_secret_access_key) ;node build' ]

  svelte-staging:
    image: ---
    networks:
      - traefik
      - postgres
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.svelte-staging.rule=Host(`staging.---.com`)"
      - "traefik.http.routers.svelte-staging.entrypoints=websecure"
      - "traefik.http.routers.svelte-staging.tls.certresolver=letsencrypt"
      - "traefik.http.services.svelte-staging.loadbalancer.server.port=3000"
    secrets:
      - source: staging_db_url
        target: db_url
        uid: '103'
        gid: '103'
        mode: 0440
    entrypoint: [ '/bin/sh', '-c', 'export DATABASE_URL=$$(cat /run/secrets/db_url) ; node build' ]

secrets:
  staging_db_url:
    external: true
  prod_db_url:
    external: true
  prod_aws_access_key_id:
    external: true
  prod_aws_secret_access_key:
    external: true

networks:
  traefik:
    external: true
  postgres:
    external: true
Wildcard Test 
$ dig test.-----.com

; <<>> DiG 9.10.6 <<>> test.----.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30008
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;test.---.com.	IN	A

;; ANSWER SECTION:
test.---.com. 300	IN	A	***.**.**.**

;; Query time: 161 msec
;; SERVER: 10.1.0.1#53(10.1.0.1)
;; WHEN: Thu Jul 06 10:44:45 CEST 2023
;; MSG SIZE  rcvd: 70
Logs on boot 
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=info msg="Configuration loaded from flags."
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=info msg="Traefik version 2.10.1 built on 2023-04-27T14:52:35Z"
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"web\":{\"address\":\":80\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{\"redirections\":{\"entryPoint\":{\"to\":\"websecure\",\"scheme\":\"https\",\"permanent\":true,\"priority\":2147483646}}},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"websecure\":{\"address\":\":443\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{\"tls\":{\"certResolver\":\"letsencrypt\",\"domains\":[{\"main\":\"c----------book.com\",\"sans\":[\"*.c----------book.com\"]}]}},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}}},\"providers\":{\"providersThrottleDuration\":\"2s\",\"docker\":{\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"network\":\"traefik\",\"swarmModeRefreshSeconds\":\"15s\"}},\"api\":{\"dashboard\":true},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"},\"certificatesResolvers\":{\"letsencrypt\":{\"acme\":{\"email\":\"------------@gmail.com\",\"caServer\":\"https://acme-v02.api.letsencrypt.org/directory\",\"storage\":\"/letsencrypt/acme.json\",\"keyType\":\"RSA4096\",\"certificatesDuration\":2160,\"dnsChallenge\":{\"provider\":\"route53\"}}}}}"
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=info msg="Starting provider aggregator aggregator.ProviderAggregator"
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Starting TCP Server" entryPointName=websecure
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Starting TCP Server" entryPointName=web
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=info msg="Starting provider *traefik.Provider"
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="*traefik.Provider provider configuration: {}"
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=info msg="Starting provider *docker.Provider"
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="*docker.Provider provider configuration: {\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"network\":\"traefik\",\"swarmModeRefreshSeconds\":\"15s\"}"
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"web-to-websecure\":{\"entryPoints\":[\"web\"],\"middlewares\":[\"redirect-web-to-websecure\"],\"service\":\"noop@internal\",\"rule\":\"HostRegexp(`{host:.+}`)\",\"priority\":2147483646}},\"services\":{\"api\":{},\"dashboard\":{},\"noop\":{}},\"middlewares\":{\"redirect-web-to-websecure\":{\"redirectScheme\":{\"scheme\":\"https\",\"port\":\"443\",\"permanent\":true}}},\"models\":{\"websecure\":{\"tls\":{\"certResolver\":\"letsencrypt\",\"domains\":[{\"main\":\"c----------book.com\",\"sans\":[\"*.c----------book.com\"]}]}}},\"serversTransports\":{\"default\":{\"maxIdleConnsPerHost\":200}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=internal
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=info msg="Starting provider *acme.ChallengeTLSALPN"
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="*acme.ChallengeTLSALPN provider configuration: {}"
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=info msg="Starting provider *acme.Provider"
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="*acme.Provider provider configuration: {\"email\":\"------------@gmail.com\",\"caServer\":\"https://acme-v02.api.letsencrypt.org/directory\",\"storage\":\"/letsencrypt/acme.json\",\"keyType\":\"RSA4096\",\"certificatesDuration\":2160,\"dnsChallenge\":{\"provider\":\"route53\"},\"ResolverName\":\"letsencrypt\",\"store\":{},\"TLSChallengeProvider\":{},\"HTTPChallengeProvider\":{}}"
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Attempt to renew certificates \"720h0m0s\" before expiry and check every \"24h0m0s\"" providerName=letsencrypt.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory"
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=info msg="Testing certificate renew..." ACME CA="https://acme-v02.api.letsencrypt.org/directory" providerName=letsencrypt.acme
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Configuration received: {\"http\":{},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=letsencrypt.acme
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Provider connection established with docker 24.0.1 (API 1.43)" providerName=docker
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Filtering disabled container" container=db-postgres-1-3c6aqk9yczfq5as274jwh78ft-fd1e969597454730940e83c1c9be2f79ea92e86b71ec29db6af593a44e22115c providerName=docker
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"http-catchall\":{\"entryPoints\":[\"web\"],\"middlewares\":[\"wwwtohttps\"],\"service\":\"proxy-traefik-1gtmr000176d2cp2u2v6phtyv-oqhfw9ficbpnsp6yslz4xyoc4\",\"rule\":\"HostRegexp(`{host:(www\\\\.)?.+}`)\"},\"svelte\":{\"entryPoints\":[\"websecure\"],\"service\":\"svelte\",\"rule\":\"Host(`c----------book.com`)\",\"tls\":{\"certResolver\":\"letsencrypt\"}},\"svelte-staging\":{\"entryPoints\":[\"websecure\"],\"service\":\"svelte-staging\",\"rule\":\"Host(`staging.c----------book.com`)\",\"tls\":{\"certResolver\":\"letsencrypt\"}},\"wwwsecure-catchall\":{\"entryPoints\":[\"websecure\"],\"middlewares\":[\"wwwtohttps\"],\"service\":\"proxy-traefik-1gtmr000176d2cp2u2v6phtyv-oqhfw9ficbpnsp6yslz4xyoc4\",\"rule\":\"HostRegexp(`{host:(www\\\\.).+}`)\",\"tls\":{}}},\"services\":{\"proxy-traefik-1gtmr000176d2cp2u2v6phtyv-oqhfw9ficbpnsp6yslz4xyoc4\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.1.183:80\"}],\"passHostHeader\":true}},\"svelte\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.1.181:3000\"}],\"passHostHeader\":true}},\"svelte-staging\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.1.178:3000\"}],\"passHostHeader\":true}}},\"middlewares\":{\"wwwtohttps\":{\"redirectRegex\":{\"regex\":\"^https?://(?:www\\\\.)?(.+)\",\"replacement\":\"https://${1}\",\"permanent\":true}}}},\"tcp\":{},\"udp\":{}}" providerName=docker
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Added outgoing tracing middleware noop@internal" entryPointName=web routerName=web-to-websecure@internal middlewareName=tracing middlewareType=TracingForwarder
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Creating middleware" routerName=web-to-websecure@internal middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme entryPointName=web
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Setting up redirection to https 443" entryPointName=web routerName=web-to-websecure@internal middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=web
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Adding certificate for domain(s) staging.c----------book.com"
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Adding certificate for domain(s) c----------book.com"
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Creating middleware" serviceName=proxy-traefik-1gtmr000176d2cp2u2v6phtyv-oqhfw9ficbpnsp6yslz4xyoc4 entryPointName=web routerName=http-catchall@docker middlewareType=Pipelining middlewareName=pipelining
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Creating load-balancer" serviceName=proxy-traefik-1gtmr000176d2cp2u2v6phtyv-oqhfw9ficbpnsp6yslz4xyoc4 entryPointName=web routerName=http-catchall@docker
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Creating server 0 http://10.0.1.183:80" serverName=0 entryPointName=web routerName=http-catchall@docker serviceName=proxy-traefik-1gtmr000176d2cp2u2v6phtyv-oqhfw9ficbpnsp6yslz4xyoc4
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="child http://10.0.1.183:80 now UP"
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Propagating new UP status"
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Added outgoing tracing middleware proxy-traefik-1gtmr000176d2cp2u2v6phtyv-oqhfw9ficbpnsp6yslz4xyoc4" routerName=http-catchall@docker middlewareName=tracing middlewareType=TracingForwarder entryPointName=web
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Creating middleware" middlewareName=wwwtohttps@docker middlewareType=RedirectRegex entryPointName=web routerName=http-catchall@docker
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Setting up redirection from ^https?://(?:www\\.)?(.+) to https://${1}" entryPointName=web routerName=http-catchall@docker middlewareName=wwwtohttps@docker middlewareType=RedirectRegex
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Adding tracing to middleware" routerName=http-catchall@docker middlewareName=wwwtohttps@docker entryPointName=web
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Added outgoing tracing middleware noop@internal" middlewareName=tracing middlewareType=TracingForwarder entryPointName=web routerName=web-to-websecure@internal
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Creating middleware" entryPointName=web routerName=web-to-websecure@internal middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Setting up redirection to https 443" entryPointName=web routerName=web-to-websecure@internal middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=pipelining middlewareType=Pipelining serviceName=proxy-traefik-1gtmr000176d2cp2u2v6phtyv-oqhfw9ficbpnsp6yslz4xyoc4 routerName=wwwsecure-catchall@docker
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Creating load-balancer" routerName=wwwsecure-catchall@docker entryPointName=websecure serviceName=proxy-traefik-1gtmr000176d2cp2u2v6phtyv-oqhfw9ficbpnsp6yslz4xyoc4
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Creating server 0 http://10.0.1.183:80" serviceName=proxy-traefik-1gtmr000176d2cp2u2v6phtyv-oqhfw9ficbpnsp6yslz4xyoc4 routerName=wwwsecure-catchall@docker serverName=0 entryPointName=websecure
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="child http://10.0.1.183:80 now UP"
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Propagating new UP status"
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Added outgoing tracing middleware proxy-traefik-1gtmr000176d2cp2u2v6phtyv-oqhfw9ficbpnsp6yslz4xyoc4" entryPointName=websecure routerName=wwwsecure-catchall@docker middlewareName=tracing middlewareType=TracingForwarder
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Creating middleware" middlewareName=wwwtohttps@docker middlewareType=RedirectRegex entryPointName=websecure routerName=wwwsecure-catchall@docker
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Setting up redirection from ^https?://(?:www\\.)?(.+) to https://${1}" entryPointName=websecure routerName=wwwsecure-catchall@docker middlewareName=wwwtohttps@docker middlewareType=RedirectRegex
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Adding tracing to middleware" routerName=wwwsecure-catchall@docker entryPointName=websecure middlewareName=wwwtohttps@docker
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Creating middleware" entryPointName=websecure serviceName=svelte middlewareName=pipelining middlewareType=Pipelining routerName=svelte@docker
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Creating load-balancer" serviceName=svelte routerName=svelte@docker entryPointName=websecure
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Creating server 0 http://10.0.1.181:3000" serviceName=svelte serverName=0 routerName=svelte@docker entryPointName=websecure
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="child http://10.0.1.181:3000 now UP"
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Propagating new UP status"
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Added outgoing tracing middleware svelte" routerName=svelte@docker middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Creating middleware" serviceName=svelte-staging middlewareName=pipelining middlewareType=Pipelining entryPointName=websecure routerName=svelte-staging@docker
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=svelte-staging@docker serviceName=svelte-staging
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Creating server 0 http://10.0.1.178:3000" routerName=svelte-staging@docker serviceName=svelte-staging serverName=0 entryPointName=websecure
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="child http://10.0.1.178:3000 now UP"
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Propagating new UP status"
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Added outgoing tracing middleware svelte-staging" middlewareType=TracingForwarder entryPointName=websecure routerName=svelte-staging@docker middlewareName=tracing
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=websecure middlewareName=traefik-internal-recovery
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=warning msg="No domain found in rule HostRegexp(`{host:(www\\.).+}`), the TLS options applied for this router will depend on the SNI of each request" entryPointName=websecure routerName=wwwsecure-catchall@docker
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Adding route for c----------book.com with TLS options default" entryPointName=websecure
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Adding route for staging.c----------book.com with TLS options default" entryPointName=websecure
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Trying to challenge certificate for domain [c----------book.com] found in HostSNI rule" rule="Host(`c----------book.com`)" ACME CA="https://acme-v02.api.letsencrypt.org/directory" providerName=letsencrypt.acme routerName=svelte@docker
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Trying to challenge certificate for domain [staging.c----------book.com] found in HostSNI rule" providerName=letsencrypt.acme routerName=svelte-staging@docker rule="Host(`staging.c----------book.com`)" ACME CA="https://acme-v02.api.letsencrypt.org/directory"
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Looking for provided certificate(s) to validate [\"staging.c----------book.com\"]..." ACME CA="https://acme-v02.api.letsencrypt.org/directory" providerName=letsencrypt.acme routerName=svelte-staging@docker rule="Host(`staging.c----------book.com`)"
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="No ACME certificate generation required for domains [\"staging.c----------book.com\"]." routerName=svelte-staging@docker rule="Host(`staging.c----------book.com`)" ACME CA="https://acme-v02.api.letsencrypt.org/directory" providerName=letsencrypt.acme
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="Looking for provided certificate(s) to validate [\"c----------book.com\"]..." routerName=svelte@docker rule="Host(`c----------book.com`)" ACME CA="https://acme-v02.api.letsencrypt.org/directory" providerName=letsencrypt.acme
proxy_traefik.0.oqhfw9ficbpn@eu-central-2    | time="2023-07-06T08:34:29Z" level=debug msg="No ACME certificate generation required for domains [\"c----------book.com\"]." rule="Host(`c----------book.com`)" ACME CA="https://acme-v02.api.letsencrypt.org/directory" providerName=letsencrypt.acme routerName=svelte@docker
Static config object from logs 
{
   "global":{
      "checkNewVersion":true
   },
   "serversTransport":{
      "maxIdleConnsPerHost":200
   },
   "entryPoints":{
      "web":{
         "address":":80",
         "transport":{
            "lifeCycle":{
               "graceTimeOut":"10s"
            },
            "respondingTimeouts":{
               "idleTimeout":"3m0s"
            }
         },
         "forwardedHeaders":{
            
         },
         "http":{
            "redirections":{
               "entryPoint":{
                  "to":"websecure",
                  "scheme":"https",
                  "permanent":true,
                  "priority":2147483646
               }
            }
         },
         "http2":{
            "maxConcurrentStreams":250
         },
         "udp":{
            "timeout":"3s"
         }
      },
      "websecure":{
         "address":":443",
         "transport":{
            "lifeCycle":{
               "graceTimeOut":"10s"
            },
            "respondingTimeouts":{
               "idleTimeout":"3m0s"
            }
         },
         "forwardedHeaders":{
            
         },
         "http":{
            "tls":{
               "certResolver":"letsencrypt",
               "domains":[
                  {
                     "main":"c----------book.com",
                     "sans":[
                        "*.c----------book.com"
                     ]
                  }
               ]
            }
         },
         "http2":{
            "maxConcurrentStreams":250
         },
         "udp":{
            "timeout":"3s"
         }
      }
   },
   "providers":{
      "providersThrottleDuration":"2s",
      "docker":{
         "watch":true,
         "endpoint":"unix:///var/run/docker.sock",
         "defaultRule":"Host(`{{ normalize .Name }}`)",
         "network":"traefik",
         "swarmModeRefreshSeconds":"15s"
      }
   },
   "api":{
      "dashboard":true
   },
   "log":{
      "level":"DEBUG",
      "format":"common"
   },
   "certificatesResolvers":{
      "letsencrypt":{
         "acme":{
            "email":"------------@gmail.com",
            "caServer":"https://acme-v02.api.letsencrypt.org/directory",
            "storage":"/letsencrypt/acme.json",
            "keyType":"RSA4096",
            "certificatesDuration":2160,
            "dnsChallenge":{
               "provider":"route53"
            }
         }
      }
   }
}
4

I recommend to review the Traefik Swarm doc again. Traefik itself needs to be constraint to manager nodes, swarmMode needs to be enabled, labels need to go below deploy section.

And clustered (multi-instance Traefik) LetsEncrypt is not supported in Traefik Community Edition, see discussions in this community

PS: Nice to see Svelte :slight_smile:

5

Docker is in swarm mode, but at the moment I only have one node at the moment. I have changed the traefik service to have the labels in the deploy section as well as specifying to run on manager only

    deploy:
      placement:
        constraints:
          - "node.role==manager"
      labels:
        traefik.enable: "true"
        # Global redirection: http to https
        traefik.http.routers.http-catchall.rule: HostRegexp(`{host:(www\.)?.+}`)
        traefik.http.routers.http-catchall.entrypoints: web
        traefik.http.routers.http-catchall.middlewares: wwwtohttps
        # Global redirection: https (www.) to https
        traefik.http.routers.wwwsecure-catchall.rule: HostRegexp(`{host:(www\.).+}`)
        traefik.http.routers.wwwsecure-catchall.entrypoints: websecure
        traefik.http.routers.wwwsecure-catchall.tls: "true"
        traefik.http.routers.wwwsecure-catchall.middlewares: wwwtohttps
        # middleware: http(s)://(www.) to  https://
        traefik.http.middlewares.wwwtohttps.redirectregex.regex: ^https?://(?:www\.)?(.+)
        traefik.http.middlewares.wwwtohttps.redirectregex.replacement: https://$${1}
        traefik.http.middlewares.wwwtohttps.redirectregex.permanent: "true"

I have enabled swarm mode

    command:
      - "--providers.docker"
      - "--providers.docker.swarmMode"

And the services now have labels moved as well

  svelte-prod:
    image: ---
    networks:
      - traefik
      - postgres
    deploy:
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.svelte.rule=Host(`---.com`)"
        - "traefik.http.routers.svelte.entrypoints=websecure"
        - "traefik.http.routers.svelte.tls.certresolver=letsencrypt"
        - "traefik.http.services.svelte.loadbalancer.server.port=3000"

I have also removed the DNS entries other than ---.com and *.---.com.

I went in and removed the acme.json file and restarted Traefik. It got certs but there is still no entry for * and Safari also still shows no entry for *.

I'm liking Svelte so far! But have been running into weird issues in production to be honest...

6

Do you use docker stack deploy?

7

Yes I deploy with docker stack deploy -c stacks/proxy/docker-compose.yml proxy

8

What does Traefik debug log say, what’s inside acme.json?

9

acme.json has the same entries as I posted before.

Here is my debug log. I tried to load the page at 2023-07-07T08:47:24Z and got bad certificates to test it.

After the actions at 2023-07-07T08:47:33Z the certificates were loading correctly.

10

So it works now? LetsEncrypt cert validation takes some seconds during startup.

11

It has always worked, but I'm not sure if it is working properly.

It seems to me like it is generating certs for each subdomain rather than a cert for the wildcard and using that for each subdomain. Maybe I am misunderstanding how it works, but I have checked with a friend who shows a wildcard cert being received/used in Safari for their site/traefik setup. My site doesn't.

12

You could try to add tls.domain.main/sans also to your router.

Compare your friends config with yours.

13

I just caught this issue in action with debug logs. I have compared with my friend and they look identical. I don't know if this is an issue with wildcard config at this point.

Below is me refreshing the page at which firefox was showing an invalid certificate a few times. I then click "continue anyway" and once I get through and apparently hit a couple 499s the cert is fine...

I find it suspicious that it is searching for a cert for www.---.com... but idk

proxy_traefik.1.zxqi6dgirhza@cn-eu-central-2    | time="2023-07-11T16:20:58Z" level=debug msg="Serving default certificate for request: \"www.---.com\""
proxy_traefik.1.zxqi6dgirhza@cn-eu-central-2    | time="2023-07-11T16:20:58Z" level=debug msg="http: TLS handshake error from ---:35430: remote error: tls: bad certificate"
proxy_traefik.1.zxqi6dgirhza@cn-eu-central-2    | time="2023-07-11T16:21:07Z" level=debug msg="Serving default certificate for request: \"www.----.com\""
proxy_traefik.1.zxqi6dgirhza@cn-eu-central-2    | time="2023-07-11T16:21:07Z" level=debug msg="http: TLS handshake error from ----:35431: remote error: tls: bad certificate"
proxy_traefik.1.zxqi6dgirhza@cn-eu-central-2    | time="2023-07-11T16:21:08Z" level=debug msg="Serving default certificate for request: \"www.----.com\""
proxy_traefik.1.zxqi6dgirhza@cn-eu-central-2    | time="2023-07-11T16:21:08Z" level=debug msg="http: TLS handshake error from ----:35432: remote error: tls: bad certificate"
proxy_traefik.1.zxqi6dgirhza@cn-eu-central-2    | time="2023-07-11T16:21:10Z" level=debug msg="Filtering disabled container" providerName=docker container=db-postgres-3c6aqk9yczfq5as274jwh78ft
proxy_traefik.1.zxqi6dgirhza@cn-eu-central-2    | time="2023-07-11T16:21:10Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"http-catchall\":{\"entryPoints\":[\"web\"],\"middlewares\":[\"wwwtohttps\"],\"service\":\"traefik\",\"rule\":\"HostRegexp(`{host:(www\\\\.)?.+}`)\"},\"svelte\":{\"entryPoints\":[\"websecure\"],\"service\":\"svelte\",\"rule\":\"Host(`---.com`)\",\"tls\":{\"certResolver\":\"letsencrypt\"}},\"svelte-staging\":{\"entryPoints\":[\"websecure\"],\"service\":\"svelte-staging\",\"rule\":\"Host(`staging.---.com`)\",\"tls\":{\"certResolver\":\"letsencrypt\"}},\"wwwsecure-catchall\":{\"entryPoints\":[\"websecure\"],\"middlewares\":[\"wwwtohttps\"],\"service\":\"traefik\",\"rule\":\"HostRegexp(`{host:(www\\\\.).+}`)\",\"tls\":{}}},\"services\":{\"svelte\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.1.222:3000\"}],\"passHostHeader\":true}},\"svelte-staging\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.1.221:3000\"}],\"passHostHeader\":true}},\"traefik\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.1.226:8080\"}],\"passHostHeader\":true}}},\"middlewares\":{\"wwwtohttps\":{\"redirectRegex\":{\"regex\":\"^https?://(?:www\\\\.)?(.+)\",\"replacement\":\"https://${1}\",\"permanent\":true}}}},\"tcp\":{},\"udp\":{}}" providerName=docker
proxy_traefik.1.zxqi6dgirhza@cn-eu-central-2    | time="2023-07-11T16:21:10Z" level=debug msg="Skipping unchanged configuration." providerName=docker
proxy_traefik.1.zxqi6dgirhza@cn-eu-central-2    | time="2023-07-11T16:21:16Z" level=debug msg="Serving default certificate for request: \"www.---.com\""
proxy_traefik.1.zxqi6dgirhza@cn-eu-central-2    | time="2023-07-11T16:21:17Z" level=debug msg="'499 Client Closed Request' caused by: context canceled"
proxy_traefik.1.zxqi6dgirhza@cn-eu-central-2    | time="2023-07-11T16:21:17Z" level=debug msg="'499 Client Closed Request' caused by: context canceled"
14

What is your friend doing differently? Different DNS provider, no wildcard?

Did you check the Route53 doc?

15

Different cloud provider and DNS provider, he has wildcard certs showing up in acme.json and in Safari. Yes that is the doc I used to set it up... but if there are any errors with that part of the process I would expect to see something showing up from Traefik right?

16

I'll try specifying the AWS config variables manually instead of just the file

EDIT: nevermind I see that I had mounted the config file then set AWS_SHARED_CREDENTIALS_FILE which is in that doc