In my traefik/letsencrypt setup which worked fine for quite some time traefik without any changes started returning traefik default certificate.
I am a bit puzzled because in my docker-compose I use a specific version of traefik (2.2.1) - so it can't be because of traefik update.
Any ideas what could it be and how to fix that? Thanks a lot!
curl https://BAR.com -vs
* Trying 45.XXX.XXX.XXX:443...
* TCP_NODELAY set
* Connected to BAR.com (45.XXX.XXX.XXX) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
Hello @yolkhovyy
Would you please share the configuration that is being used? You can also include log files that could help to investigate the issue.
Thank you,
@jakubhajek Thanks for your quick response, here it is:
traefik:
image: traefik:v2.2.1
configs:
- source: traefik_users
target: /users
mode: 444
ports:
- "80:80"
- "443:443"
command:
- --api.insecure=false
- --api.dashboard=true # see https://docs.traefik.io/v2.0/operations/dashboard/#secure-mode for how to secure the dashboard
# - --api.debug=true # enable additional endpoints for debugging and profiling
- --accesslog=true
- --accesslog.filepath=/var/log/traefik/traefik.log
- --accesslog.bufferingsize=100
- --log.level=DEBUG
- --providers.file.filename=/config/dynamic.toml
- --providers.file.watch=true
- --providers.docker=true
- --providers.docker.endpoint=unix:///var/run/docker.sock
- --providers.docker.swarmMode=true
- --providers.docker.exposedByDefault=false
- --providers.docker.network=proxy
- --metrics
- --metrics.prometheus.buckets=0.1,0.3,1.2,5.0
- --entrypoints.ep-http.address=:80
- --entrypoints.ep-https.address=:443
#- --certificatesresolvers.certresolver.acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory
- --certificatesresolvers.certresolver.acme.caServer=https://acme-v02.api.letsencrypt.org/directory
- --certificatesresolvers.certresolver.acme.tlschallenge=true
- --certificatesresolvers.certresolver.acme.email=FOO.BAR@gmail.com
- --certificatesresolvers.certresolver.acme.storage=/letsencrypt/acme.json
volumes:
- traefik:/config
- log:/var/log
- letsencrypt:/letsencrypt
- /var/run/docker.sock:/var/run/docker.sock
networks:
- proxy
deploy:
replicas: 1
restart_policy:
condition: any
delay: 3s
placement:
constraints:
# - node.role == manager
- node.hostname == rpi4
labels:
- traefik.enable=true
# Generic redirect to https
- traefik.http.middlewares.mw-redirect-to-https.redirectscheme.scheme=https
- traefik.http.middlewares.mw-redirect-to-https.redirectscheme.permanent=true
- traefik.http.routers.r-redirect-to-https.rule=hostregexp(`{host:.+}`)
- traefik.http.routers.r-redirect-to-https.entrypoints=ep-http #,ep-portainer
- traefik.http.routers.r-redirect-to-https.middlewares=mw-redirect-to-https
# Secure Traefik
- traefik.http.middlewares.mw-traefik-auth.basicauth.usersfile=/users
- traefik.http.middlewares.mw-traefik-auth.basicauth.realm=traefik
- traefik.http.routers.r-traefik.entrypoints=ep-https
- traefik.http.routers.r-traefik.rule=Host(`traefik.BAZ.com`) && (PathPrefix(`/api/`) || PathPrefix(`/dashboard/`))
- traefik.http.routers.r-traefik.tls=true
- traefik.http.routers.r-traefik.tls.certresolver=certresolver
- traefik.http.routers.r-traefik.middlewares=mw-traefik-auth
- traefik.http.routers.r-traefik.service=api@internal
- traefik.http.services.dummy-service.loadbalancer.server.port=1357
There are tons of these errors:
management_traefik.1.q4nzziq6120h@rpi4 | time="2022-01-13T14:47:15Z" level=error msg="Unable to obtain ACME certificate for domains \"traefik.BAZ.com\": unable to generate a certificate for the domains [traefik.BAZ.com]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/, url: " providerName=certresolver.acme routerName=r-traefik@docker rule="Host(`traefik.BAZ.com`) && (PathPrefix(`/api/`) || PathPrefix(`/dashboard/`))"
I have tried letsencrypt staging endpoint - I think the errors disappeared, but was still the traefik default certificate returned.
Just found this error in my logs - HTTP 400 Bad Request:
Unable to obtain ACME certificate for domains \"traefik.bar.com\": unable to generate a certificate for the domains [traefik.bar.com]: error: one or more domains had a problem:\n[traefik.bar.com] acme: error: 400 :: urn:ietf:params:acme:error:tls :: remote error: tls: internal error\n"
I found a solution that worked - my setup runs in a Docker Swarm cluster, and I recreated from scratch the node where Traefik ran, specifically updated/upgraded the OS, completely purged/reinstalled docker - I think there was something with it, I saw somewhere people described similar issue and their conclusion was that the docker's ip tables were messed up. I am not sure it was the same in my case, but I saw the errors below in my logs - that might have had to do with docker's ip tables I guess. All works after the node re-creation - I hope this info will help other people too.
time="2022-01-20T08:05:41Z" level=error msg="close tcp [::]:443: use of closed network connection" entryPointName=ep-https
time="2022-01-20T08:05:41Z" level=error msg="close tcp [::]:8080: use of closed network connection" entryPointName=traefik
time="2022-01-20T08:05:41Z" level=error msg="close tcp [::]:80: use of closed network connection" entryPointName=ep-http
