LetsEncrypt JSON file is not being stored

I got LetsEncrypt working now, but I noticed something was off in that when I restart the server it appears to recreate the certificate again. The commands I am using to configure traefik are:

  - --certificatesresolvers.default.acme.httpchallenge.entrypoint=http
  - --certificatesResolvers.default.acme.email=support@trajano.net
  - --certificatesResolvers.default.acme.storage=/letsencrypt/acme.json

I have this volumes block on the service

volumes:
  - letsencrypt:/letsencrypt

To correspond to the following in the top level compose file:

volumes:
  letsencrypt:

I see the volume when I do docker volume ls but when I try to view it's contents it is empty using this command

docker run -it -v edge_letsencrypt:/mnt bash ls -la //mnt

If you try to save the cert on your local machine, does it save?

$ sudo touch ~/acme.json
$ sudo chmod 600 ~/acme.json

Then mount it with:

 volumes
  - ~/acme.json:/letsencrypt/acme.json

After you restart traefik if you sudo cat ~/acme.json it should show you the contents of the cert. Once you troubleshoot your cert issue you can look at putting it in a docker volume as you're trying to do if you wish.

Also, I'm not sure if you need this, but according to the docs you should add:

- "--certificatesresolvers.default.acme.httpchallenge=true"

And I see you're already doing this, but as a reference for others, the cert resolver also needs to be referenced in the routers:

- "traefik.http.routers.< Router-1 >.tls.certresolver=default"

If that doesn't help you troubleshoot the issue, could you share your compose script and traefik.toml?

1 Like
version: "3.7"
services:
  traefik:
    image: traefik:v2.0.0-rc2
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
      - target: 8080
        published: 44444
        protocol: tcp
        mode: host
    command:
      - --global.checknewversion=false
      - --global.sendanonymoususage=false

      - --accesslog
      - --api
      - --ping

      - --certificatesresolvers.default.acme.httpchallenge.entrypoint=http
      - --certificatesResolvers.default.acme.email=support@trajano.net
      - --certificatesResolvers.default.acme.storage=/letsencrypt/acme.json

      - --certificatesresolvers.staging.acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory
      - --certificatesresolvers.staging.acme.httpchallenge.entrypoint=http
      - --certificatesResolvers.staging.acme.email=support@trajano.net
      - --certificatesResolvers.staging.acme.storage=/letsencrypt/staging.json

      - --entrypoints.http.address=:80
      - --entrypoints.https.address=:443

      - --providers.docker.endpoint=tcp://daemon:2375
      - --providers.docker.exposedByDefault=false
      - --providers.docker.swarmMode=true

      - --tracing.zipkin.httpEndpoint=http://zipkin:9411/api/v2/spans
      - --tracing.zipkin.samespan=true

      # - --log.level=INFO
      - --log.level=DEBUG
    # healthcheck:
    #   test: ./traefik healtcheck --ping
    volumes:
      - letsencrypt:/letsencrypt
    networks:
      - traefik
      - internal
      - default
    deploy:
      mode: global
      resources:
        limits:
          memory: 64M
        reservations:
          memory: 16M
  daemon:
    image: alpine/socat
    command: tcp-listen:2375,fork,reuseaddr unix-connect:/var/run/docker.sock
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - internal
    deploy:
      placement:
        constraints:
          - node.role == manager
      resources:
        limits:
          cpus: '0.5'
          memory: 32M
  ping:
    image: alpine/socat
    command: tcp-listen:8080,fork,reuseaddr tcp:traefik:8080
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 32M
      labels:
        - traefik.enable=true
        - traefik.http.routers.ping.rule=Host(`trajano.net`) && Path(`/ping`)
        - traefik.http.routers.ping.entryPoints=https
        - traefik.http.routers.ping.middlewares=default
        - traefik.http.routers.ping.tls.certresolver=default
        - traefik.http.routers.ping.service=ping
        - traefik.http.services.ping.loadbalancer.server.port=8080

        - traefik.http.routers.insecure.rule=HostRegexp(`trajano.net`, `{subdomain:[a-z]+}.trajano.net`)
        - traefik.http.routers.insecure.entryPoints=http
        - traefik.http.routers.insecure.middlewares=https-only

        - traefik.http.middlewares.https-only.redirectscheme.scheme=https
        - traefik.http.middlewares.https-only.redirectscheme.permanent=true

        - traefik.http.middlewares.default.chain.middlewares=compress-all
        - traefik.http.middlewares.compress-all.compress=true
    networks:
      - traefik
  zipkin:
    image: openzipkin/zipkin
    networks:
      - internal
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 128M
  zookeeper:
    image: zookeeper:3.4.13
    networks:
      - internal
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 64M
networks:
  internal:
    internal: true
    attachable: false
  traefik:
    name: traefik
    internal: true
    attachable: true
volumes:
  letsencrypt:

Before I make those alterations you pointed out

I'm on Windows so there's no chmod my alternation to do that was to create a file and share it

    volumes:
      # - letsencrypt:/letsencrypt
      - /d/s/acme.json:/letsencrypt/acme.json

regardless as I sort of expected... file is empty
image

Certificate does get generated though

Since Traefik has removed any form of "shell" in the docker image, it makes is harder to debug what's going on.

Even a docker cp yields no results

image

after doing some low level docker checks (i.e. going to the machine and searching for acme.json in /var/lib/docker/overlay2 it appears that Traefik is putting the file in /acme.json rather than /letsencrypt/acme.json

Actually looking at it a bit further, it seems to ignore the value for

- --certificatesResolvers.default.acme.storage="staging.json"

Because I only see acme.json and not even staging.json above.

Hmm Windows machine so no chmod and no sudo? I've only used docker on linux so cant help there. It sounds like you need to troubleshoot why traefik isn't picking up your commands. It seems with v2 that if there are any issues with your config it will break traefik, so I would suggest starting with only the traefik service and get that working first. Start with just one router, one cert, one entrypoint, and check to see if it's working, saving the cert where you want etc, and go from there.

Just upgraded to rc3 the problem is still there in that it does not put it in the storage folder specified.

Upgraded to rc4, issue still persists though the "bot" had already closed the ticket. :man_shrugging: can't really do much about this.

Issue is still persisting after the 2,0.0 release

Anyway @CVJoint , I just went into the machine's file system and added the folder as I had surmised, it does not create the file in the folder specified by storage

However, your technique of using the file itself does work, it still does not resolve the fact that this https://docs.traefik.io/https/acme/#in-a-file is not correct.

Nor this part in https://docs.traefik.io/reference/static-configuration/cli/

--certificatesresolvers.<name>.acme.storage :
Storage to use. (Default: acme.json )

I eventually got this working with the environment variables configuration.

  TRAEFIK_CERTIFICATESRESOLVERS_DEFAULT_ACME_EMAIL: support@trajano.net
  TRAEFIK_CERTIFICATESRESOLVERS_DEFAULT_ACME_HTTPCHALLENGE_ENTRYPOINT: http
  TRAEFIK_CERTIFICATESRESOLVERS_DEFAULT_ACME_STORAGE: /etc/traefik/acme/default.json

It's just a problem related to case:

  - --certificatesresolvers.default.acme.httpchallenge.entrypoint=http
  - --certificatesResolvers.default.acme.email=support@trajano.net
  - --certificatesResolvers.default.acme.storage=/letsencrypt/acme.json

certificatesresolvers vs certificatesResolvers => you need to use the same case

  - --certificatesresolvers.default.acme.httpchallenge.entrypoint=http
  - --certificatesresolvers.default.acme.email=support@trajano.net
  - --certificatesresolvers.default.acme.storage=/letsencrypt/acme.json

or

  - --certificatesResolvers.default.acme.httpchallenge.entrypoint=http
  - --certificatesResolvers.default.acme.email=support@trajano.net
  - --certificatesResolvers.default.acme.storage=/letsencrypt/acme.json

The flags are case insensitive but the case on one element must be identical.

Can we get a warning on the logs when something like that is detected.

In the next path release, the problem will be fixed: https://github.com/containous/traefik/pull/5438

The case will no longer need to be consistent.