Help migrating to v2.0 wildcard ACME and non-existent middleware

[madcatsu]

Hi,

I'm also relatively new to Traefik v2 and still trying to get my head around how to migrate my v1.x config. Following the docs for v2, I've created a new YAML file for my Traefik container and added a bunch of new labels to my Docker Compose file.

Initially I had a problem with declaring a common middleware across my containers which defines a bunch of HTTPS headers, redirecting HTTP-to-HTTPS and securing the /api endpoint which I've since resolved. I use the DNS-01 ACME challenge for LE certificate generation and have provided my DNSimple OAUTH token and API base URL as env vars to my Traefik container.

Now I have these in place, my Traefik container outputs a couple of errors I just can't seem to get my head around resolving.

1. level=error msg="Unable to obtain ACME certificate for domains \"vacuum.sub.domain.com\": unable to generate a certificate for the domains ... : acme: Error -> One or more domains had a problem:\n vacuum.sub.domain.com acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: No TXT record found at _acme-challenge.sub.domain.com, url: \n" providerName=default.acme rule="Host(vacuum.sub.domain.com)" routerName=vacuum

2. time="2019-09-25T05:16:17+02:00" level=error msg="middleware \"default@docker\" does not exist" entryPointName=web-secure routerName=vacuum@docker

Would really appreciate a bit of help understanding how I can resolve these and simplify my config.

log:
  level: "ERROR"
api:
  dashboard: true
http:
  routers:
    api:
      rule: PathPrefix(`/api`) || PathPrefix(`/dashboard`)
      service: api@internal
      middlewares:
        - auth
  middlewares:
    auth:
      basicAuth:
        users:
        - "redacted:redacted"
    default:
      headers:
        sslRedirect: true
        sslHost: sub.domain.com
        stsSeconds: 315360000
        stsIncludeSubdomains: true
        browserXssFilter: true
        contentTypeNosniff: true
        forceSTSHeader: true
        frameDeny: true
      redirectScheme:
        scheme: https
providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    watch: true
entrypoints:
  web:
    address: ":80"
  web-secure:
    address: ":443"
certificatesResolvers:
  default:
    acme:
      email: service-certs@sub.domain.com
      storage: acme.json
      dnsChallenge:
        provider: dnsimple
        delayBeforeCheck: 0

docker-compose.yml

My goal here is to serve the API endpoint externally with basic auth for now, and publish all services as children of the sub-domain.

[zespri]

Hello there,

do you know about the difference between static and dynamic configuration in traefik?

[madcatsu]

Still getting me head around that and I think I might be best to use a static configuration for the middleware I'm calling 'default' which has the SSH header options.

So in my mind, I would create a file provider in traefik.yml like this:

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    watch: true
  file:
    filename: middleware.yml

that references something like middleware.yml and move all my relevant middleware rules there like so:

http:
  middlewares:
    default:
      headers:
        sslRedirect: true
        sslHost: sub.domain.com
        stsSeconds: 315360000
        stsIncludeSubdomains: true
        browserXssFilter: true
        contentTypeNosniff: true
        forceSTSHeader: true
        frameDeny: true
      redirectScheme:
        scheme: https

Then in Docker Compose, if I've got this correct, I would reference the middleware as:

- "traefik.http.routers.vacuum.middlewares=default@file"

Have I got that right?

[zespri]

Yep, that's sounds about right to me. Try that, and if you still get errors you cannot figure out, post here again!

I think you can also specify middleware via labels as per here but I personally would try to do what you are doing, because to me the yaml looks nicer than the corresponding labels.

[madcatsu]

Indeed, I can just see easily 20 labels per container, not so fun to maintain when if there is an option to declare a common set somewhere else as a directive.

Will try and see what turns up next

[madcatsu]

Going further down the rabbit hole it seems instead of getting out!

Now getting:

time="2019-09-25T11:06:06+02:00" level=error msg="cannot create middleware: multi-types middleware not supported, consider declaring two different pieces of middleware instead" entryPointName=web routerName=vacuum@docker

and

time="2019-09-25T11:06:07+02:00" level=error msg="cannot create middleware: multi-types middleware not supported, consider declaring two different pieces of middleware instead" routerName=vacuum@docker entryPointName=web-secure

Also getting this

time="2019-09-25T11:07:16+02:00" level=error msg="Unable to obtain ACME certificate for domains \"domain.com,*.domain.com\" : unable to generate a certificate for the domains [domain.com *.domain.com]: acme: Error -> One or more domains had a problem:\n[*.domain.com] [*.domain.com] acme: error presenting token: dnsimple: API call failed: POST https://api.dnsimple.com/v2/74286/zones/domain.com/records: 400 Zone record already exists\n[domain.com] time limit exceeded: last error: NS ns3.dnsimple.com. returned NXDOMAIN for _acme-challenge.domain.com.\n" providerName=default.acme
time="2019-09-25T11:07:16+02:00" level=error msg="Unable to obtain ACME certificate for domains \"domain.com,*.domain.com\" : unable to generate a certificate for the domains [domain.com *.domain.com]: acme: Error -> One or more domains had a problem:\n[*.domain.com] time limit exceeded: last error: NS ns1.dnsimple.com. did not return the expected TXT record [fqdn: _acme-challenge.domain.com., value: ...\n[domain.com] [domain.com] acme: error presenting token: dnsimple: API call failed: POST https://api.dnsimple.com/v2/74286/zones/domain.com/records: 400 Zone record already exists\n" providerName=default.acme

My middleware.yml looks like this:

http:
  middlewares:
    default:
      headers:
        sslRedirect: true
        sslHost: sub.domain.com
        stsSeconds: 315360000
        stsIncludeSubdomains: true
        browserXssFilter: true
        contentTypeNosniff: true
        forceSTSHeader: true
        frameDeny: true
      redirectScheme:
        scheme: https

traefik.yml is now this:

log:
  level: "ERROR"
api:
  dashboard: true
http:
  routers:
    api:
      rule: PathPrefix(`/api`) || PathPrefix(`/dashboard`)
      service: api@internal
      middlewares:
        - auth
  middlewares:
    auth:
      basicAuth:
        users:
        - "redacted:redacted"
providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    watch: true
  file:
    filename: /etc/traefik/middleware.yml
entrypoints:
  web:
    address: ":80"
  web-secure:
    address: ":443"
certificatesResolvers:
  default:
    acme:
      email: service-certs@domain.com
      storage: acme.json
      dnsChallenge:
        provider: dnsimple
        delayBeforeCheck: 0

and my updated Docker compose file now looks like this:

version: '3.7'
services:

  octopus:
    container_name: "proxy"
    image: traefik:2.0
    restart: unless-stopped
    networks:
      - frontend
      - internal
    ports:
      - 80:80
      - 443:443
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock
      - /proxy/config/:/etc/traefik
    env_file: ./proxy.env
    depends_on:
      - vacuum
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=frontend"
      - "traefik.http.routers.octopus.rule=Host(`octopus.sub.domain.com`)"
      - "traefik.http.routers.octopus.entrypoints=web, web-secure"
      - "traefik.http.routers.octopus.tls.certresolver=default"
      - "traefik.http.routers.octopus.tls.domains[0].main=domain.com"
      - "traefik.http.routers.octopus.tls.domains[0].sans=*.domain.com"
      - "traefik.http.routers.octopus.service=octopus"
      - "traefik.http.routers.octopus.middlewares=default@file"
      - "traefik.http.services.octopus.loadbalancer.server.port=8080"
      - "traefik.http.services.octopus.loadbalancer.server.scheme=http"
  
  vacuum:
    container_name: "deluge"
    image: linuxserver/deluge
    restart: unless-stopped
    networks:
      - frontend
    ports:
      - "58846:58846"
      - "58946:58946"
      - "58946:58946/udp"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /deluge/config:/config
      - /staging/downloads:/downloads
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=frontend"
      - "traefik.http.routers.vacuum.rule=Host(`vacuum.sub.domain.com`)"
      - "traefik.http.routers.vacuum.entrypoints=web, web-secure"
      - "traefik.http.routers.vacuum.tls.certresolver=default"
      - "traefik.http.routers.vacuum.tls.domains[0].main=domain.com"
      - "traefik.http.routers.vacuum.tls.domains[0].sans=*.domain.com"
      - "traefik.http.routers.vacuum.service=vacuum"
      - "traefik.http.routers.vacuum.middlewares=default@file"
      - "traefik.http.services.vacuum.loadbalancer.server.port=8112"
      - "traefik.http.services.vacuum.loadbalancer.server.scheme=http"

Definitely stuck right now and not sure where to go with this

Note the domain I'm using is just an example, the domain I'm actually using reports this error.

[ldez]

You can not create multi-types middleware, consider declaring two different pieces of middleware instead:

http:
  middlewares:

    head:
      headers:
        sslRedirect: true
        sslHost: sub.domain.com
        stsSeconds: 315360000
        stsIncludeSubdomains: true
        browserXssFilter: true
        contentTypeNosniff: true
        forceSTSHeader: true
        frameDeny: true

    foobar:
      redirectScheme:
        scheme: https

labels:
- "traefik.http.routers.vacuum.middlewares=foobar@file,headers@file"

You can also create a chain:

http:
  middlewares:

    head:
      headers:
        sslRedirect: true
        sslHost: sub.domain.com
        stsSeconds: 315360000
        stsIncludeSubdomains: true
        browserXssFilter: true
        contentTypeNosniff: true
        forceSTSHeader: true
        frameDeny: true

    foobar:
      redirectScheme:
        scheme: https

    mychain:
      chain:
        middlewares:
          - head
          - foobar

labels:
- "traefik.http.routers.vacuum.middlewares=mychain@file"

• https://docs.traefik.io/v2.0/middlewares/chain/

[madcatsu]

Looks like I'm getting somewhere @ldez

I'm now getting the default Traefik self-signed root cert and a number of my endpoints are just returning a 404 with nothing to suggest there's a problem with fetching the cert

[ldez]

time="2019-09-25T11:07:16+02:00" level=error msg="Unable to obtain ACME certificate for domains \"domain.com,*.domain.com\" : unable to generate a certificate for the domains [domain.com *.domain.com]: acme: Error -> One or more domains had a problem:\n[*.domain.com] [*.domain.com] acme: error presenting token: dnsimple: API call failed: POST https://api.dnsimple.com/v2/74286/zones/domain.com/records: 400 Zone record already exists\n[domain.com] time limit exceeded: last error: NS ns3.dnsimple.com. returned NXDOMAIN for _acme-challenge.domain.com.\n" providerName=default.acme

Non eXistent DOMAIN, so your domain (domain.com) don't exist or the A record is missing

time="2019-09-25T11:07:16+02:00" level=error msg="Unable to obtain ACME certificate for domains \"domain.com,*.domain.com\" : unable to generate a certificate for the domains [domain.com *.domain.com]: acme: Error -> One or more domains had a problem:\n[*.domain.com] time limit exceeded: last error: NS ns1.dnsimple.com. did not return the expected TXT record [fqdn: _acme-challenge.domain.com., value: ...\n[domain.com] [domain.com] acme: error presenting token: dnsimple: API call failed: POST https://api.dnsimple.com/v2/74286/zones/domain.com/records: 400 Zone record already exists\n" providerName=default.acme

You have to clean your TXT records

[ldez]

also you cannot mix dynamic configuration and static configuration in the same file:

traefik.toml:

log:
  level: "ERROR"

api:
  dashboard: true

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    watch: true
  file:
    directory: /etc/traefik/config/

entrypoints:
  web:
    address: ":80"
  web-secure:
    address: ":443"

certificatesResolvers:
  default:
    acme:
      email: service-certs@domain.com
      storage: acme.json
      dnsChallenge:
        provider: dnsimple
        delayBeforeCheck: 0

/etc/traefik/config/api.yml

http:
  routers:
    api:
      rule: PathPrefix(`/api`) || PathPrefix(`/dashboard`)
      service: api@internal
      middlewares:
        - auth
  middlewares:
    auth:
      basicAuth:
        users:
        - "redacted:redacted"

also move /etc/traefik/middleware.yml to /etc/traefik/config/middleware.yml

/proxy/config/
├── traefik.yml
└── config
    ├── api.yml
    └── middleware.yml

/etc/traefik
├── traefik.yml
└── config
    ├── api.yml
    └── middleware.yml

[madcatsu]

OK, that last point is really useful to know!! Thx

Seems I've accidentally hit the LE rate limit so I'll have to wait a bit or move to an alternate domain to continue validating any other issues.

[ldez]

I recommend to use the Let's Encrypt staging server during the test to avoid rate limit:

certificatesResolvers:
  sample:
    acme:
      # ...
      caServer: https://acme-staging-v02.api.letsencrypt.org/directory
      # ...

https://docs.traefik.io/v2.0/https/acme/#caserver

[ldez]

@madcatsu

Still getting me head around that and I think I might be best to use a static configuration for the middleware I'm calling 'default' which has the SSH header options.

The middlewares can only be define in the dynamic configuration.

dynamic configuration:

• routers
• middlewares
• services
• tls

documentation:

• Configuration Introduction | Traefik | v2.0
• File | Traefik | v2.0

static configuration:

• information about the providers to used
• logger definition
• ...

documentation:

• Configuration Introduction | Traefik | v2.0
• File | Traefik | v2.0

[madcatsu]

OK, so besides the invalid cert error now on some container endpoints, a few also just come back as a vanilla 404 from Traefik, it seems the redirect to SSL isn't being followed from the middlewares rule

[ldez]

You have mixed several concept, so I will try to show you the way:

version: '3.7'
services:

  octopus:
    container_name: "proxy"
    image: traefik:2.0
    restart: unless-stopped
    networks:
      - frontend
      - internal
    ports:
      - 80:80
      - 443:443
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock
      - /proxy/config/:/etc/traefik
    env_file: ./proxy.env
    depends_on:
      - vacuum
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=frontend"
      - "traefik.http.routers.dashboard.rule=Host(`octopus.sub.domain.com`)"
      - "traefik.http.routers.dashboard.entrypoints=web-secure"
      - "traefik.http.routers.dashboard.tls.certresolver=default"
      - "traefik.http.routers.dashboard.tls.domains[0].main=domain.com"
      - "traefik.http.routers.dashboard.tls.domains[0].sans=*.domain.com"
      - "traefik.http.routers.dashboard.service: api@internal"
      - "traefik.http.routers.dashboard.middlewares=auth@file"
  
  vacuum:
    container_name: "deluge"
    image: linuxserver/deluge
    restart: unless-stopped
    networks:
      - frontend
    ports:
      - "58846:58846"
      - "58946:58946"
      - "58946:58946/udp"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /deluge/config:/config
      - /staging/downloads:/downloads
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=frontend"
      - "traefik.http.routers.vacuum.rule=Host(`vacuum.sub.domain.com`)"
      - "traefik.http.routers.vacuum.entrypoints=web-secure"
      - "traefik.http.routers.vacuum.tls.certresolver=default"
      - "traefik.http.routers.vacuum.tls.domains[0].main=domain.com"
      - "traefik.http.routers.vacuum.tls.domains[0].sans=*.domain.com"
      - "traefik.http.routers.vacuum.service=vacuum"
      - "traefik.http.routers.vacuum.middlewares=auth@file"
      - "traefik.http.services.vacuum.loadbalancer.server.port=8112"

/proxy/config/
├── traefik.yml
└── dynamic
    ├── auth.yml
    └── globalredirect.yml

/etc/traefik
├── traefik.yml
└── dynamic
    ├── auth.yml
    └── globalredirect.yml

# /etc/traefik/traefik.yaml
## static configuration

log:
  level: "ERROR"

api: {}

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    watch: true
  file:
    directory: /etc/traefik/dynamic/

entrypoints:
  web:
    address: ":80"
  web-secure:
    address: ":443"

certificatesResolvers:
  default:
    acme:
      email: service-certs@domain.com
      storage: acme.json
      caServer: "https://acme-staging-v02.api.letsencrypt.org/directory"
      dnsChallenge:
        provider: dnsimple
        delayBeforeCheck: 0

# /etc/traefik/dynamic/auth.yaml
## dynamic configuration

# basic auth

http:
  middlewares:
    auth:
      basicAuth:
        users:
        - "redacted:redacted"

## dynamic configuration
# /etc/traefik/dynamic/globalredirect.yml

# create a global redirection to https

http:
  middlewares:
    redirecttohttps:
      redirectScheme:
        scheme: https
  
  routers:
    redirecttohttps:
      rule: "HostRegexp(`{host:.+}`)"
      entryPoints:
      - web
      middlewares:
      - redirecttohttps
      service: "noop"

  services:
    # noop service, the URL will be never called
    noop:
      loadBalancer:
        servers:
        - url: "http://192.168.0.1"

[madcatsu]

This makes much more sense now. Sorry, I wasn't really able to glean these concepts very well from existing documentation, as good as it is.

[madcatsu]

@ldez - much progress made. I can now reach all my endpoints but I'm now getting a bunch of LE errors for DNS-01 challenge.

time="2019-09-26T00:46:44+02:00" level=error msg="Unable to obtain ACME certificate for domains \"replicant.vaulted.cloud,*.replicant.vaulted.cloud\" : unable to generate a certificate for the domains [replicant.vaulted.cloud *.replicant.vaulted.cloud]: acme: Error -> One or more domains had a problem:\n[*.replicant.vaulted.cloud] [*.replicant.vaulted.cloud] acme: error presenting token: dnsimple: API call failed: POST https://api.dnsimple.com/v2/74286/zones/vaulted.cloud/records: 400 Zone record already exists\n" providerName=default.acme
time="2019-09-26T00:46:53+02:00" level=error msg="Unable to obtain ACME certificate for domains \"replicant.vaulted.cloud,*.replicant.vaulted.cloud\" : unable to generate a certificate for the domains [replicant.vaulted.cloud *.replicant.vaulted.cloud]: acme: Error -> One or more domains had a problem:\n[*.replicant.vaulted.cloud] failed to initiate challenge: acme: error: 400 :: POST :: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/10441876/qgHu7w :: urn:ietf:params:acme:error:malformed :: Unable to update challenge :: authorization must be pending, url: \n[replicant.vaulted.cloud] [replicant.vaulted.cloud] acme: error presenting token: dnsimple: API call failed: POST https://api.dnsimple.com/v2/74286/zones/vaulted.cloud/records: 400 Zone record already exists\n" providerName=default.acme
time="2019-09-26T00:47:47+02:00" level=error msg="Unable to obtain ACME certificate for domains \"replicant.vaulted.cloud,*.replicant.vaulted.cloud\" : unable to generate a certificate for the domains [replicant.vaulted.cloud *.replicant.vaulted.cloud]: acme: Error -> One or more domains had a problem:\n[replicant.vaulted.cloud] time limit exceeded: last error: NS ns2.dnsimple.com. did not return the expected TXT record [fqdn: _acme-challenge.replicant.vaulted.cloud., value: nQ8U3r94qlYs92bmRUUv0avdrNG71cp8zjC_qPtuujs]: \n" providerName=default.acme
time="2019-09-26T00:48:12+02:00" level=error msg="Unable to obtain ACME certificate for domains \"replicant.vaulted.cloud,*.replicant.vaulted.cloud\" : unable to generate a certificate for the domains [replicant.vaulted.cloud *.replicant.vaulted.cloud]: acme: Error -> One or more domains had a problem:\n[*.replicant.vaulted.cloud] time limit exceeded: last error: NS ns3.dnsimple.com. did not return the expected TXT record [fqdn: _acme-challenge.replicant.vaulted.cloud., value: RoB9OYDpQ8-alOnDH2h8YvnzQKSp2sblQjnZaWMMWJE]: \n[replicant.vaulted.cloud] time limit exceeded: last error: NS ns1.dnsimple.com. did not return the expected TXT record [fqdn: _acme-challenge.replicant.vaulted.cloud., value: ULYdn9s_Ad9UmewOqT5FjBW3_lxJwsC7NjdhHgrOZo4]: \n" providerName=default.acme
time="2019-09-26T00:48:12+02:00" level=error msg="Unable to obtain ACME certificate for domains \"replicant.vaulted.cloud,*.replicant.vaulted.cloud\" : unable to generate a certificate for the domains [replicant.vaulted.cloud *.replicant.vaulted.cloud]: acme: Error -> One or more domains had a problem:\n[*.replicant.vaulted.cloud] time limit exceeded: last error: NS ns2.dnsimple.com. did not return the expected TXT record [fqdn: _acme-challenge.replicant.vaulted.cloud., value: td6zPBqMjOcKfQ-2MdHY8pv5iD0kVqDP1YMe-jf67oQ]: \n[replicant.vaulted.cloud] time limit exceeded: last error: NS ns4.dnsimple.com. did not return the expected TXT record [fqdn: _acme-challenge.replicant.vaulted.cloud., value: Ljx833nFv1eLb8eujx1G1ImUR8wXf_0rIHETTXEZm9U]: \n" providerName=default.acme

I'm using the staging ACME endpoint and this is a domain that exists and was working for DNS-01 challenge previously in Traefik 1.7

[ldez]

@madcatsu

Zone record already exists

You have to clean your TXT records in the DNSimple admin panel.

[madcatsu]

@ldez I have 3 TXT records currently in this zone, none for _acme-challenge.replicant.vaulted.cloud

Other records are

vaulted.cloud. 3600 IN TXT "v=spf1 ... ~all"
vaulted.cloud. 3600 IN TXT "google-site-verification=..."
google._domainkey.vaulted.cloud. 3600 IN TXT ...

I can also see a bunch of TXT record deletes in my audit trail in DNSimple

Record	Delete	TXT _acme-challenge.replicant.vaulted.cloud ULYdn9s_Ad9UmewOqT5FjBW3_lxJwsC7NjdhHgrOZo4	redacted	Sep 25, 2019 22:46:42
Record	Delete	TXT _acme-challenge.replicant.vaulted.cloud nQ8U3r94qlYs92bmRUUv0avdrNG71cp8zjC_qPtuujs	redacted	Sep 25, 2019 22:46:42
Record	Delete	TXT _acme-challenge.replicant.vaulted.cloud Ljx833nFv1eLb8eujx1G1ImUR8wXf_0rIHETTXEZm9U	redacted	Sep 25, 2019 22:46:42

I think it's creating a duplicate _acme-challenge TXT record, one for the root domain (I only have A records for the apex, not CNAME's) and then attempting to create another for *.replicant.vaulted.cloud

I see https://github.com/containous/traefik/issues/5317 is already tracking this

[ldez]

The code about DNSimple in the same between v1 and v2.

So could you re-check that similar config works with the v1.7?