Traefik 2.0: Http challenge does not seem to work

Traefik Traefik v2 (latest)
1

Do you want to request a feature or report a bug?

Bug

Did you try using a 1.7.x configuration for the version 2.0?

  • [ ] Yes
  • [X] No

What did you do?

We have a k8s cluster which hosts services that our customers want to address with their own domains, of which we are not able to control the DNS records, hence we want to swap out the dns in favour of the http challenge. Using Traefik 1.7.x we got a lot of errors from acme / lego, so we switched to 2.0 GA.

Our cluster contains one master and two worker nodes. Traefik is deployed as a daemon set on both worker nodes. For starters, we want to make the http challenge work for services laying behind our company intern domain. The DNS records of this Domain are managed by Cloudflare, and each subdomain points to both worker nodes. We do not use Cloudflare's SSL, SSL is only meant to be terminated by Traefik.

Example DNS records:

A swift.cloudiety.de node0IP
A swift.cloudiety.de node1IP

The swift pod in this example is a swift-nio hello world project, listening on 8080.
We have not managed to make the http challenge work.

What did you expect to see?

A Let's Encrypt staging certificate being issued for swift.cloudiety.de

What did you see instead?

curl rejects the certificate, stating it's TRAEFIK DEFAULT CERT, so I it's not a LE staging certificate

* Rebuilt URL to: swift.cloudiety.de/
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 195.201.221.205...
* TCP_NODELAY set
* Connected to swift.cloudiety.de (195.201.221.205) port 80 (#0)
> GET / HTTP/1.1
> Host: swift.cloudiety.de
> User-Agent: curl/7.54.0
> Accept: */*
> 
< HTTP/1.1 302 Found
< Location: https://swift.cloudiety.de/
< Date: Wed, 18 Sep 2019 13:58:33 GMT
< Content-Length: 5
< Content-Type: text/plain; charset=utf-8
< 
* Ignoring the response-body
{ [5 bytes data]

100     5  100     5    0     0    115      0 --:--:-- --:--:-- --:--:--   116
* Connection #0 to host swift.cloudiety.de left intact
* Issue another request to this URL: 'https://swift.cloudiety.de/'
*   Trying 195.201.221.205...
* TCP_NODELAY set
* Connected to swift.cloudiety.de (195.201.221.205) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [224 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [58 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [852 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [300 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=TRAEFIK DEFAULT CERT
*  start date: Sep 18 13:58:32 2019 GMT
*  expire date: Sep 17 13:58:32 2020 GMT
*  issuer: CN=TRAEFIK DEFAULT CERT
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f92b2800400)
> GET / HTTP/2
> Host: swift.cloudiety.de
> User-Agent: curl/7.54.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200 
< content-type: text/plain; charset=utf-8
< content-length: 13
< date: Wed, 18 Sep 2019 13:58:33 GMT
< 
{ [13 bytes data]

100    13  100    13    0     0     86      0 --:--:-- --:--:-- --:--:--    86
* Connection #1 to host swift.cloudiety.de left intact
Hello World

Output of traefik version: (What version of Traefik are you using?)

Version:      2.0.0
Codename:     montdor
Go version:   go1.13
Built:        2019-09-16T17:35:17Z
OS/Arch:      linux/amd64

What is your environment & configuration (arguments, toml, provider, platform, ...)?

Traefik DaemonSet

apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: default
  name: traefik-ingress-controller

---
kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
  namespace: default
  name: traefik
  labels:
    app: traefik

spec:
  selector:
    matchLabels:
      app: traefik
  template:
    metadata:
      labels:
        app: traefik
    spec:
      serviceAccountName: traefik-ingress-controller
      containers:
        - name: traefik
          image: traefik:v2.0
          args:
            - --api.insecure
            - --accesslog
            - --log.level=debug
            - --entrypoints.web.Address=:80
            - --entrypoints.websecure.Address=:443
            - --providers.kubernetescrd
            - --certificatesResolvers.default.acme.httpChallenge=true
            - --certificatesresolvers.default.acme.httpChallenge.entryPoint=web
            - --certificatesresolvers.default.acme.email=devops@worldiety.de
            - --certificatesresolvers.default.acme.storage=acme.json
            # Please note that this is the staging Let's Encrypt server.
            # Once you get things working, you should remove that whole line altogether.
            - --certificatesresolvers.default.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
          ports:
            - name: web
              containerPort: 80
              hostPort: 80
            - name: websecure
              containerPort: 443
              hostPort: 443
            - name: admin
              containerPort: 8080
          securityContext:
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE

Traefik Service

apiVersion: v1
kind: Service
metadata:
  name: traefik

spec:
  ports:
    - protocol: TCP
      name: web
      port: 80
    - protocol: TCP
      name: admin
      port: 8080
    - protocol: TCP
      name: websecure
      port: 443
  selector:
    app: traefik

---
apiVersion: v1
kind: Service
metadata:
  name: whoami

spec:
  ports:
    - protocol: TCP
      name: web
      port: 80
  selector:
    app: whoami

IngressRoutes

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: swift-ingress-http
  namespace: default
spec:
  entryPoints:
    - web
  routes:
  - kind: Rule
    priority: 10
    match: Host(`swift.cloudiety.de`)
    middlewares:
      - name: https-redirect
    services:
    - name: swift-nio-service
      port: 8080
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: swift-ingress-https
  namespace: default
spec:
  entryPoints:
    - websecure
  routes:
  - kind: Rule
    priority: 11
    match: Host(`swift.cloudiety.de`)
    middlewares:
      - name: https-redirect
    services:
    - name: swift-nio-service
      port: 8080
  tls: {}

swift nio deployment

---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: swift-nio-test
  namespace: default
  labels:
    k8s-app: swift-nio-test
spec:
  replicas: 2
  selector:
    matchLabels:
      k8s-app: swift-nio-test
  template:
    metadata:
      labels:
        k8s-app: swift-nio-test
        name: swift-nio-test
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - image: juka/swift-nio-test
          name: swift-nio
          ports:
            - name: http
              containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: swift-nio-service
  namespace: default
spec:
  selector:
    k8s-app: swift-nio-test
  ports:
    - name: web
      port: 8080
      targetPort: 8080

If applicable, please paste the log output in DEBUG level (--log.level=DEBUG switch)

Logs from Traefik instance on node 0, filtered by acme

$ kubectl  logs traefik-t9dpz | grep -i acme
time="2019-09-18T12:51:21Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"traefik\":{\"address\":\":8080\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{}},\"web\":{\"address\":\":80\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{}},\"websecure\":{\"address\":\":443\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{}}},\"providers\":{\"providersThrottleDuration\":2000000000,\"kubernetesCRD\":{}},\"api\":{\"insecure\":true,\"dashboard\":true},\"log\":{\"level\":\"debug\",\"format\":\"common\"},\"accessLog\":{\"format\":\"common\",\"filters\":{},\"fields\":{\"defaultMode\":\"keep\",\"headers\":{\"defaultMode\":\"drop\"}}},\"certificatesResolvers\":{\"default\":{\"acme\":{\"email\":\"devops@worldiety.de\",\"caServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"storage\":\"acme.json\",\"keyType\":\"RSA4096\",\"httpChallenge\":{\"entryPoint\":\"web\"}}}}}"
time="2019-09-18T12:51:21Z" level=info msg="Starting provider *acme.Provider {\"email\":\"devops@worldiety.de\",\"caServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"storage\":\"acme.json\",\"keyType\":\"RSA4096\",\"httpChallenge\":{\"entryPoint\":\"web\"},\"ResolverName\":\"default\",\"store\":{},\"ChallengeStore\":{}}"
time="2019-09-18T12:51:21Z" level=info msg="Testing certificate renew..." providerName=default.acme
time="2019-09-18T12:51:21Z" level=debug msg="Configuration received from provider default.acme: {\"http\":{},\"tls\":{}}" providerName=default.acme

Logs from Traefik instance on node 1, filtered by acme

$ kubectl  logs traefik-kvmnh | grep -i acme
time="2019-09-18T12:51:21Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"traefik\":{\"address\":\":8080\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{}},\"web\":{\"address\":\":80\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{}},\"websecure\":{\"address\":\":443\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{}}},\"providers\":{\"providersThrottleDuration\":2000000000,\"kubernetesCRD\":{}},\"api\":{\"insecure\":true,\"dashboard\":true},\"log\":{\"level\":\"debug\",\"format\":\"common\"},\"accessLog\":{\"format\":\"common\",\"filters\":{},\"fields\":{\"defaultMode\":\"keep\",\"headers\":{\"defaultMode\":\"drop\"}}},\"certificatesResolvers\":{\"default\":{\"acme\":{\"email\":\"devops@worldiety.de\",\"caServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"storage\":\"acme.json\",\"keyType\":\"RSA4096\",\"httpChallenge\":{\"entryPoint\":\"web\"}}}}}"
time="2019-09-18T12:51:21Z" level=info msg="Starting provider *acme.Provider {\"email\":\"devops@worldiety.de\",\"caServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"storage\":\"acme.json\",\"keyType\":\"RSA4096\",\"httpChallenge\":{\"entryPoint\":\"web\"},\"ResolverName\":\"default\",\"store\":{},\"ChallengeStore\":{}}"
time="2019-09-18T12:51:21Z" level=info msg="Testing certificate renew..." providerName=default.acme
time="2019-09-18T12:51:21Z" level=debug msg="Configuration received from provider default.acme: {\"http\":{},\"tls\":{}}" providerName=default.acme
2

Hello,

you have to define the cert resolver on the router:

  tls:
    certResolver: default

https://docs.traefik.io/user-guides/crd-acme/#traefik-routers

3

On which of my IngressRoute definitions do I have to add that?

The docs say the following:

If you need to define the same route for both HTTP and HTTPS requests, you will need to define two different routers: one with the tls section, one without.

So, I defined two IngressRoute definitions:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: swift-ingress-http
  namespace: default
spec:
  entryPoints:
    - web
  routes:
  - kind: Rule
    priority: 10
    match: Host(`swift.cloudiety.de`)
    middlewares:
      - name: https-redirect
    services:
    - name: swift-nio-service
      port: 8080

This first one is thought to be responsible to take traffic on port 80 and redirect it to 443 with the middleware https-redirect

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: swift-ingress-https
  namespace: default
spec:
  entryPoints:
    - websecure
  routes:
  - kind: Rule
    priority: 11
    match: Host(`swift.cloudiety.de`)
    middlewares:
      - name: https-redirect
    services:
    - name: swift-nio-service
      port: 8080
  tls: {}

The second one listens on 443 and terminates TLS, and forwards the traffic to my service.

Regarding the tls section I saw the following happen:

  • If I add certResolver: default to the first one, connections to http://swift.cloudiety.de will receive a 404, connections to https:// will get the DEFAULT_TRAEFIK_CERT
  • If i add it to the second one, connections to http:// still redirect to https:// and will then receive the DEFAULT_TRAEFIK_CERT
  • If i omit certResolver: default from the tls section of the second IngressRoute (as in my original post), I get the same behaviour as if I added it (see bullet point above)

So none of the above mentioned configurations seem to work