Traefik Not Updating Certs to Prod Certs After Deleting Staging Certs

tonydattolo · March 27, 2024, 3:13pm

I recently switched off of staging certificates to go to production. I deleted the volume mount, and switched the storage location and caserver. I deleted the service and re-deployed it with docker stack deploy. I also changed the name of the resolver just in-case that was what was causing the issue.

Why is it still recognizing the now deleted staging certs?

  traefik:
    image: traefik:v2.11
    command:
      # https://doc.traefik.io/traefik/reference/static-configuration/cli/
      - --api.dashboard=true
      - --log.level=DEBUG
      - --providers.docker=true
      - --providers.docker.endpoint=unix:///var/run/docker.sock
      - --providers.docker.swarmmode=true
      - --providers.docker.exposedbydefault=false  # Don't expose every service by default
      - --providers.docker.network=traefik-network
      - --entrypoints.web
      - --entrypoints.web.address=:80  # Define HTTP entry point
      - --entrypoints.web.http.redirections.entrypoint.to=websecure
      - --entryPoints.web.http.redirections.entrypoint.scheme=https
      - --entrypoints.websecure
      - --entrypoints.websecure.address=:443  # Define HTTPS entry point
      - --entrypoints.websecure.http.tls.certresolver=myresolver
      - --certificatesresolvers.myresolver
      - --certificatesresolvers.myresolver.acme.httpchallenge=true
      - --certificatesresolvers.myresolver.acme.httpchallenge.entrypoint=web
      - --certificatesresolvers.myresolver.acme.tlschallenge=true
      - --certificatesresolvers.myresolver.acme.email=myemail@email.com
      - --certificatesresolvers.myresolver.acme.caserver=https://acme-v02.api.letsencrypt.org/directory # prod
      - --certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json # prod
      # - --certificatesresolvers.myresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory # staging
      # - --certificatesresolvers.myresolver.acme.storage=/letsencrypt/staging/acme.json # staging
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "~/mnt/data/traefik/acme.json:/letsencrypt/acme.json"
      # - "~/mnt/data/traefik/staging/acme.json:/letsencrypt/staging/acme.json"
    deploy:
      placement:
        constraints:
          - node.role == manager
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.traefik-http.rule=Host(`traefik.example.com`)"
        - "traefik.http.routers.traefik-http.middlewares=authtraefik"
        - "traefik.http.routers.traefik-http.service=api@internal"
        - "traefik.http.routers.traefik-http.entrypoints=web"
        - "traefik.http.routers.traefik_http.tls.domains[0].main=example.com"
        - "traefik.http.routers.traefik-https.rule=Host(`traefik.example.com`)"
        - "traefik.http.routers.traefik-https.middlewares=authtraefik"
        - "traefik.http.routers.traefik-https.entrypoints=websecure"
        - "traefik.http.routers.traefik-https.service=api@internal"
        - "traefik.http.routers.traefik-https.tls=true"
        - "traefik.http.routers.traefik_https.tls.domains[0].main=example.com"
        - "traefik.http.routers.traefik-https.tls.certresolver=myresolver"
        - "traefik.http.services.traefik.loadbalancer.server.port=8080"
        - "traefik.http.middlewares.authtraefik.basicauth.users=admin:$$passwordhashhere"
      restart_policy:
        condition: on-failure
        delay: 30s
        max_attempts: 3
        window: 120s
    networks:
      - traefik-network

bluepuma77 · March 27, 2024, 4:13pm

You can only enable a single challenge for LetsEncrypt, not multiple.

It seems you are using Docker Swarm, note that Traefik CE LetsEncrypt is not cluster-enabled, so you can only run a single instance.

Check Traefik debug log for acme and/or error.

tonydattolo · March 27, 2024, 4:51pm

Looks like maybe the volumes weren't being deleted for some reason. Used portainer to delete the container with the "delete non-persistent volumes" option checked off.

time="2024-03-27T16:50:02Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=web middlewareName=traefik-internal-recovery

time="2024-03-27T16:50:02Z" level=debug msg="Creating middleware" entryPointName=web routerName=django-http@docker serviceName=django middlewareName=pipelining middlewareType=Pipelining

time="2024-03-27T16:50:02Z" level=debug msg="Creating load-balancer" serviceName=django entryPointName=web routerName=django-http@docker

time="2024-03-27T16:50:02Z" level=debug msg="Creating server 0 http://10.0.1.212:8000" serviceName=django entryPointName=web routerName=django-http@docker serverName=0

time="2024-03-27T16:50:02Z" level=debug msg="child http://10.0.1.212:8000 now UP"

time="2024-03-27T16:50:02Z" level=debug msg="Propagating new UP status"

time="2024-03-27T16:50:02Z" level=debug msg="Added outgoing tracing middleware django" routerName=django-http@docker middlewareName=tracing middlewareType=TracingForwarder entryPointName=web

time="2024-03-27T16:50:02Z" level=debug msg="Creating middleware" middlewareName=pipelining middlewareType=Pipelining entryPointName=web routerName=traefik_http@docker serviceName=traefik

time="2024-03-27T16:50:02Z" level=debug msg="Creating load-balancer" entryPointName=web routerName=traefik_http@docker serviceName=traefik

time="2024-03-27T16:50:02Z" level=debug msg="Creating server 0 http://10.0.1.223:8080" serverName=0 entryPointName=web routerName=traefik_http@docker serviceName=traefik

time="2024-03-27T16:50:02Z" level=debug msg="child http://10.0.1.223:8080 now UP"

time="2024-03-27T16:50:02Z" level=debug msg="Propagating new UP status"

time="2024-03-27T16:50:02Z" level=debug msg="Added outgoing tracing middleware traefik" entryPointName=web routerName=traefik_http@docker middlewareType=TracingForwarder middlewareName=tracing

time="2024-03-27T16:50:02Z" level=debug msg="Creating middleware" middlewareName=pipelining middlewareType=Pipelining entryPointName=web routerName=traefik_https@docker serviceName=traefik

time="2024-03-27T16:50:02Z" level=debug msg="Creating load-balancer" entryPointName=web routerName=traefik_https@docker serviceName=traefik

time="2024-03-27T16:50:02Z" level=debug msg="Creating server 0 http://10.0.1.223:8080" serviceName=traefik entryPointName=web routerName=traefik_https@docker serverName=0

time="2024-03-27T16:50:02Z" level=debug msg="child http://10.0.1.223:8080 now UP"

time="2024-03-27T16:50:02Z" level=debug msg="Propagating new UP status"

time="2024-03-27T16:50:02Z" level=debug msg="Added outgoing tracing middleware traefik" middlewareType=TracingForwarder entryPointName=web routerName=traefik_https@docker middlewareName=tracing

time="2024-03-27T16:50:02Z" level=debug msg="Creating middleware" entryPointName=web middlewareType=Recovery middlewareName=traefik-internal-recovery

time="2024-03-27T16:50:02Z" level=debug msg="Creating middleware" entryPointName=websecure routerName=websecure-traefik_http@docker serviceName=traefik middlewareName=pipelining middlewareType=Pipelining

time="2024-03-27T16:50:02Z" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=websecure-traefik_http@docker serviceName=traefik

time="2024-03-27T16:50:02Z" level=debug msg="Creating server 0 http://10.0.1.223:8080" entryPointName=websecure routerName=websecure-traefik_http@docker serviceName=traefik serverName=0

time="2024-03-27T16:50:02Z" level=debug msg="child http://10.0.1.223:8080 now UP"

time="2024-03-27T16:50:02Z" level=debug msg="Propagating new UP status"

time="2024-03-27T16:50:02Z" level=debug msg="Added outgoing tracing middleware traefik" routerName=websecure-traefik_http@docker middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure

time="2024-03-27T16:50:02Z" level=debug msg="Added outgoing tracing middleware api@internal" middlewareType=TracingForwarder entryPointName=websecure routerName=traefik-https@docker middlewareName=tracing

time="2024-03-27T16:50:02Z" level=debug msg="Creating middleware" entryPointName=websecure routerName=traefik-https@docker middlewareName=authtraefik@docker middlewareType=BasicAuth

time="2024-03-27T16:50:02Z" level=debug msg="Adding tracing to middleware" entryPointName=websecure routerName=traefik-https@docker middlewareName=authtraefik@docker

time="2024-03-27T16:50:02Z" level=debug msg="Creating middleware" routerName=portainer-https@docker serviceName=portainer middlewareName=pipelining middlewareType=Pipelining entryPointName=websecure

time="2024-03-27T16:50:02Z" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=portainer-https@docker serviceName=portainer

time="2024-03-27T16:50:02Z" level=debug msg="Creating server 0 http://10.0.1.204:9000" entryPointName=websecure routerName=portainer-https@docker serviceName=portainer serverName=0

time="2024-03-27T16:50:02Z" level=debug msg="child http://10.0.1.204:9000 now UP"

time="2024-03-27T16:50:02Z" level=debug msg="Propagating new UP status"

time="2024-03-27T16:50:02Z" level=debug msg="Added outgoing tracing middleware portainer" middlewareType=TracingForwarder middlewareName=tracing entryPointName=websecure routerName=portainer-https@docker

time="2024-03-27T16:50:02Z" level=debug msg="Creating middleware" serviceName=traefik middlewareName=pipelining middlewareType=Pipelining entryPointName=websecure routerName=websecure-traefik_https@docker

time="2024-03-27T16:50:02Z" level=debug msg="Creating load-balancer" routerName=websecure-traefik_https@docker serviceName=traefik entryPointName=websecure

time="2024-03-27T16:50:02Z" level=debug msg="Creating server 0 http://10.0.1.223:8080" entryPointName=websecure routerName=websecure-traefik_https@docker serviceName=traefik serverName=0

time="2024-03-27T16:50:02Z" level=debug msg="child http://10.0.1.223:8080 now UP"

time="2024-03-27T16:50:02Z" level=debug msg="Propagating new UP status"

time="2024-03-27T16:50:02Z" level=debug msg="Added outgoing tracing middleware traefik" middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure routerName=websecure-traefik_https@docker

time="2024-03-27T16:50:02Z" level=debug msg="Creating middleware" routerName=django-https@docker serviceName=django middlewareName=pipelining middlewareType=Pipelining entryPointName=websecure

time="2024-03-27T16:50:02Z" level=debug msg="Creating load-balancer" routerName=django-https@docker serviceName=django entryPointName=websecure

time="2024-03-27T16:50:02Z" level=debug msg="Creating server 0 http://10.0.1.212:8000" entryPointName=websecure routerName=django-https@docker serviceName=django serverName=0

time="2024-03-27T16:50:02Z" level=debug msg="child http://10.0.1.212:8000 now UP"

time="2024-03-27T16:50:02Z" level=debug msg="Propagating new UP status"

time="2024-03-27T16:50:02Z" level=debug msg="Added outgoing tracing middleware django" middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure routerName=django-https@docker

time="2024-03-27T16:50:02Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=websecure

time="2024-03-27T16:50:02Z" level=debug msg="Adding route for api.example.ai with TLS options default" entryPointName=web

time="2024-03-27T16:50:02Z" level=debug msg="Adding route for app-traefik with TLS options default" entryPointName=web

time="2024-03-27T16:50:02Z" level=debug msg="Adding route for app-traefik with TLS options default" entryPointName=websecure

time="2024-03-27T16:50:02Z" level=debug msg="Adding route for api.example.ai with TLS options default" entryPointName=websecure

time="2024-03-27T16:50:02Z" level=debug msg="Adding route for traefik.example.ai with TLS options default" entryPointName=websecure

time="2024-03-27T16:50:02Z" level=debug msg="Adding route for portainer.example.ai with TLS options default" entryPointName=websecure

time="2024-03-27T16:50:02Z" level=error msg="the router traefik-https@docker uses a non-existent resolver: myresolver"

time="2024-03-27T16:50:02Z" level=error msg="the router portainer-https@docker uses a non-existent resolver: myresolver"

time="2024-03-27T16:50:02Z" level=error msg="the router django-https@docker uses a non-existent resolver: myresolver"

It's saying it doesn't recognize our resolver and bad TLS handshake request

time="2024-03-27T17:02:50Z" level=debug msg="Serving default certificate for request: \"traefik.example.ai\""

time="2024-03-27T17:02:50Z" level=debug msg="http: TLS handshake error from 38.125.230.172:42188: remote error: tls: bad certificate"

tonydattolo · March 27, 2024, 6:41pm

for resolution:

if we let traefik manage the acme.json file completely as opposed to creating it and adding it directly to the volume, that works. I guess if there's a blank file there it gets messed up?

before:

 - "~/mnt/data/traefik/acme.json:/letsencrypt/acme.json"

after:

 - "~/mnt/data/traefik:/letsencrypt"

system · March 30, 2024, 6:42pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Unable to reset certs from staging to production Traefik v2 docker	1	892	May 8, 2023
Can get staging cert but not production cert Traefik v3 (latest) docker , letsencrypt-acme	2	153	July 16, 2024
Traefik won't update certs Traefik v2 docker-swarm	7	3123	March 19, 2020
[fixed] Docker Swarm + LetsEncrypt with just labels and command line (no TOML) not working Traefik v2 docker-swarm , letsencrypt-acme	1	744	September 5, 2019
OK. I Give Up... Converting V1.7 to V2.1 Traefik v2 docker-swarm	4	1165	December 4, 2019

Traefik Not Updating Certs to Prod Certs After Deleting Staging Certs

Related topics