V2 https example

tcurdt · October 30, 2019, 2:12am

I am a little lost trying to get the following working:

a single http service on port 5151 exposed as https on 443
a self signed cert when trying locally, letsencrypt for production
protected path /foo by a client cert (if possible)

I started with the following:

version: '3.7'
services:
  web:
    image: traefik:2.0.4
    restart: unless-stopped
    command:
      - "--api.insecure=true"
      - "--global.sendAnonymousUsage=false"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=true"
      - "--entryPoints.https.address=:443"
      # - "--certificatesResolvers.letsencrypt.acme.httpChallenge=true"
      # - "--certificatesResolvers.letsencrypt.acme.httpChallenge.entryPoint=https"
      # - "--certificatesresolvers.letsencrypt.acme.email=XXXXXXXXX"
      # - "--certificatesresolvers.letsencrypt.acme.storage=/opt/traefik/acme.json"
    ports:
      - 443:443
      - 127.0.0.1:8080:8080
    networks:
      - local
    depends_on:
      - backend
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./certs:/certs:ro
  backend:
    image: containous/whoami # just for testing
    restart: unless-stopped
    ports:
      - 5151:80 # real backend will have 5151:5151
    labels:
      - "traefik.port=5151"
      - "traefik.http.routers.backend-https.rule=Host(`machine.test`)"
      - "traefik.http.routers.backend-https.entrypoints=https"
      # - "traefik.http.routers.backend-https.tls.certresolver=letsencrypt"
networks:
  local:

Where I am struggling:

howto add a self signed cert
verify letsencrypt is working like this
howto require a client cert for /foo

Can anyone point me at least into the right direction?
Thanks!

ldez · October 30, 2019, 8:07am

Hello,

I recommend to read https://blog.containo.us/traefik-2-0-docker-101-fc2893944b9d

tcurdt · October 30, 2019, 10:23am

Super helpful article - thanks!
Unfortunately it does not cover my open questions.
I am still not sure how to specify the following via labels:

[[tls.certificates]]
  certFile = "/path/to/domain.cert"
  keyFile = "/path/to/domain.key"

[tls.options]
  [tls.options.default]
    [tls.options.default.clientAuth]
      caFiles = ["tests/clientca1.crt", "tests/clientca2.crt"]
      clientAuthType = "RequireAndVerifyClientCert"

and how to make the client cert only apply for a certain path.

ldez · October 30, 2019, 10:32am

Restriction

In the above example, we've used the file provider to handle these definitions. It is the only available method to configure the certificates (as well as the options and the stores).

https://docs.traefik.io/v2.0/https/tls/

tcurdt · October 30, 2019, 10:48am

I read this as "it's not possible to pass as parameters/labels - it must be defined in a file". IIUC I can only use --providers.file.filename to specify that instead?

But that will not help me to apply the client cert only to path /foo.

tcurdt · November 1, 2019, 10:18am

Is this correct? The paths to the certs must be specified in a file?

tcurdt · November 1, 2019, 10:23am

For letsencrypt support I came up with

version: '3.7'
services:
  web:
    image: traefik:2.0.4
    restart: unless-stopped
    command:
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=true"
      - "--entryPoints.https.address=:443"
      - "--certificatesresolvers.letsencrypt.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
      - "--certificatesresolvers.letsencrypt.acme.email=me@foo.com"
      - "--certificatesresolvers.letsencrypt.acme.storage=/vol/acme.json"
      - "--certificatesResolvers.letsencrypt.acme.tlsChallenge=true"
    ports:
      - 443:443
    networks:
      - local
    depends_on:
      - backend
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./vol/traefik:/vol

  backend:
    image: containous/whoami
    command:
      - "--port=5678"
    restart: unless-stopped
    labels:
      - "traefik.http.routers.backend-https.rule=Host(`foo.com`)"
      - "traefik.http.routers.backend-https.entrypoints=https"
      - "traefik.http.services.backend-https.loadbalancer.server.port=5678"
      - "traefik.http.routers.backend-https.tls.certresolver=letsencrypt"

networks:
  local:

but not I am not sure what's wrong with that config. It's giving me a 404.
Any advice? Thanks!

tcurdt · November 1, 2019, 1:06pm

Update with the DEBUG log:

version: '3.7'
services:
  web:
    image: traefik:2.0.4
    restart: unless-stopped
    command:
      - "--log.level=DEBUG"
      - "--providers.docker=true"
      - "--entryPoints.https.address=:443"
    ports:
      - 443:443
    networks:
      - local
    depends_on:
      - backend
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
  backend:
    image: containous/whoami
    command:
      - "--port=5678"
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.backend.entrypoints=https"
      - "traefik.http.routers.backend.rule=HostRegexp(`{host:.+}`)"
      - "traefik.http.services.backend.loadbalancer.server.port=5678"
networks:
  local:

Produces:

backend_1  | Starting up on port 5678
web_1      | time="2019-11-01T12:57:53Z" level=info msg="Configuration loaded from flags."
web_1      | time="2019-11-01T12:57:53Z" level=info msg="Traefik version 2.0.4 built on 2019-10-28T20:23:57Z"
web_1      | time="2019-11-01T12:57:53Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"https\":{\"address\":\":443\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{}}},\"providers\":{\"providersThrottleDuration\":2000000000,\"docker\":{\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"exposedByDefault\":true,\"swarmModeRefreshSeconds\":15000000000}},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"}}"
web_1      | time="2019-11-01T12:57:53Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/v2.0/contributing/data-collection/\n"
web_1      | time="2019-11-01T12:57:53Z" level=debug msg="No default certificate, generating one"
web_1      | time="2019-11-01T12:57:53Z" level=info msg="Starting provider aggregator.ProviderAggregator {}"
web_1      | time="2019-11-01T12:57:53Z" level=debug msg="Start TCP Server" entryPointName=https
web_1      | time="2019-11-01T12:57:53Z" level=info msg="Starting provider *docker.Provider {\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"exposedByDefault\":true,\"swarmModeRefreshSeconds\":15000000000}"
web_1      | time="2019-11-01T12:57:53Z" level=debug msg="Provider connection established with docker 19.03.2 (API 1.40)" providerName=docker
web_1      | time="2019-11-01T12:57:53Z" level=debug msg="Configuration received from provider docker: {\"http\":{\"routers\":{\"backend\":{\"entryPoints\":[\"https\"],\"service\":\"backend\",\"rule\":\"HostRegexp(`{host:.+}`)\"},\"web-traefik\":{\"service\":\"web-traefik\",\"rule\":\"Host(`web-traefik`)\"}},\"services\":{\"backend\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.20.0.2:5678\"}],\"passHostHeader\":true}},\"web-traefik\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.21.0.2:80\"}],\"passHostHeader\":true}}}},\"tcp\":{}}" providerName=docker
web_1      | time="2019-11-01T12:57:53Z" level=debug msg="No entryPoint defined for this router, using the default one(s) instead: [https]" routerName=web-traefik@docker
web_1      | time="2019-11-01T12:57:53Z" level=debug msg="Creating middleware" middlewareName=pipelining middlewareType=Pipelining entryPointName=https routerName=web-traefik@docker serviceName=web-traefik
web_1      | time="2019-11-01T12:57:53Z" level=debug msg="Creating load-balancer" entryPointName=https routerName=web-traefik@docker serviceName=web-traefik
web_1      | time="2019-11-01T12:57:53Z" level=debug msg="Creating server 0 http://172.21.0.2:80" routerName=web-traefik@docker serviceName=web-traefik serverName=0 entryPointName=https
web_1      | time="2019-11-01T12:57:53Z" level=debug msg="Added outgoing tracing middleware web-traefik" routerName=web-traefik@docker middlewareName=tracing middlewareType=TracingForwarder entryPointName=https
web_1      | time="2019-11-01T12:57:53Z" level=debug msg="Creating middleware" serviceName=backend middlewareName=pipelining middlewareType=Pipelining entryPointName=https routerName=backend@docker
web_1      | time="2019-11-01T12:57:53Z" level=debug msg="Creating load-balancer" serviceName=backend entryPointName=https routerName=backend@docker
web_1      | time="2019-11-01T12:57:53Z" level=debug msg="Creating server 0 http://172.20.0.2:5678" routerName=backend@docker serviceName=backend serverName=0 entryPointName=https
web_1      | time="2019-11-01T12:57:53Z" level=debug msg="Added outgoing tracing middleware backend" entryPointName=https routerName=backend@docker middlewareType=TracingForwarder middlewareName=tracing
web_1      | time="2019-11-01T12:57:53Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=https middlewareName=traefik-internal-recovery
web_1      | time="2019-11-01T12:57:53Z" level=debug msg="No default certificate, generating one"

$ curl -k -I https://localhost
HTTP/2 404 
content-type: text/plain; charset=utf-8
x-content-type-options: nosniff
content-length: 19
date: Fri, 01 Nov 2019 12:58:05 GMT

web_1      | time="2019-11-01T12:58:05Z" level=debug msg="Serving default certificate for request: \"localhost\""

Please - any pointers? Thanks.

PS: Why is web-traefik being created?

ldez · November 2, 2019, 11:04pm

It's related to https://docs.traefik.io/v2.0/providers/docker/#exposedbydefault

A simple TLS example:

version: '3.7'
services:
  web:
    image: traefik:v2.0.4
    command:
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
      - --api=true
      - --log.level=INFO
      - --providers.docker=true
      - --providers.docker.exposedbydefault=false
    ports:
      - 80:80
      - 443:443
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    labels:
      - "traefik.enable=true"

      # Dashboard
      - "traefik.http.routers.traefik.rule=Host(`traefik.docker.localhost`)"
      - "traefik.http.routers.traefik.service=api@internal"
      - "traefik.http.routers.traefik.entrypoints=websecure"
      - "traefik.http.routers.traefik.tls=true"

      # global redirect HTTPS
      - "traefik.http.routers.catchall.rule=HostRegexp(`{host:.+}`)"
      - "traefik.http.routers.catchall.entrypoints=web"
      - "traefik.http.routers.catchall.middlewares=redirecthttps"
      
      # redirect HTTPS
      - "traefik.http.middlewares.redirecthttps.redirectscheme.scheme=https"

  backend:
    image: containous/whoami
    command:
      - "--port=5678"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.backend.rule=Host(`whoami.docker.localhost`)"
      - "traefik.http.routers.backend.entrypoints=websecure"
      - "traefik.http.routers.backend.tls=true"
      - "traefik.http.services.backend.loadbalancer.server.port=5678"

tcurdt · November 2, 2019, 11:48pm

Thanks for the example, @ldez - but that's giving me the very same 404.

I've adopted it to strip out the dashboard and port 80 and to make it even simpler - but are still getting a 404.

version: '3.7'
services:
  web:
    image: traefik:2.0.4
    restart: unless-stopped
    command:
      - "--entryPoints.https.address=:443"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--log.level=DEBUG"
    ports:
      - 443:443
    depends_on:
      - backend
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro

  backend:
    image: containous/whoami
    command:
      - "--port=5678"
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.backend.rule=HostRegexp(`{host:.+}`)"
      - "traefik.http.routers.backend.entrypoints=https"
      - "traefik.http.routers.backend.tls=true"
      - "traefik.http.services.backend.loadbalancer.server.port=5678"

Something is really off here.

ldez · November 3, 2019, 12:13am

Your example is working:

version: '3.7'
services:
  web:
    image: traefik:2.0.4
    command:
      - "--entryPoints.https.address=:443"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--log.level=DEBUG"
    ports:
      - 443:443
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro

  backend:
    image: containous/whoami
    command:
      - "--port=5678"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.backend.rule=HostRegexp(`{host:.+}`)"
      - "traefik.http.routers.backend.entrypoints=https"
      - "traefik.http.routers.backend.tls=true"
      - "traefik.http.services.backend.loadbalancer.server.port=5678"

https://localhost -> whoami

$ curl -k https://localhost

Hostname: cea44e30ae28
IP: 127.0.0.1
IP: 172.23.0.2
RemoteAddr: 172.23.0.3:55778
GET / HTTP/1.1
Host: localhost
User-Agent: curl/7.66.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 172.23.0.1
X-Forwarded-Host: localhost
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: d50b7420c3ca
X-Real-Ip: 172.23.0.1

I recommend to clean your networks: docker prune network

Are you using swarm?

tcurdt · November 3, 2019, 12:50am

Odd. It's not working here:

$ docker network prune
$ docker-compose up
$ curl -k https://localhost
404 page not found
$ docker -v
Docker version 19.03.2, build 6a30dfc
$ docker-compose -v
docker-compose version 1.24.1, build 4667896b

ldez · November 3, 2019, 12:57am

$ docker -v
Docker version 19.03.4-ce, build 9013bf583a
$ docker-compose -v
docker-compose version 1.24.1, build unknown

$ drill localhost

;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 36499
;; flags: qr aa rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 
;; QUESTION SECTION:
;; localhost.	IN	A

;; ANSWER SECTION:
localhost.	0	IN	A	127.0.0.1

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 4 msec
;; SERVER: 192.168.1.1
;; WHEN: Sun Nov  3 01:55:23 2019
;; MSG SIZE  rcvd: 43

$ curl -k -H Host:localhost https://127.0.0.1

Hostname: cea44e30ae28
IP: 127.0.0.1
IP: 172.23.0.2
RemoteAddr: 172.23.0.3:57072
GET / HTTP/1.1
Host: localhost
User-Agent: curl/7.66.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 172.23.0.1
X-Forwarded-Host: localhost
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: d50b7420c3ca
X-Real-Ip: 172.23.0.1

$ curl -k -H Host:test https://127.0.0.1

Hostname: cea44e30ae28
IP: 127.0.0.1
IP: 172.23.0.2
RemoteAddr: 172.23.0.3:57072
GET / HTTP/1.1
Host: test
User-Agent: curl/7.66.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 172.23.0.1
X-Forwarded-Host: test
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: d50b7420c3ca
X-Real-Ip: 172.23.0.1

tcurdt · November 3, 2019, 1:06am

Given that it is https and the only reachable when the containers are up it really must be traefik returning the 404 (on my machine).

$ curl -k -H Host:localhost https://127.0.0.1
404 page not found

Plus I see this request in the logs with level DEBUG:

web_1      | time="2019-11-03T01:04:06Z" level=debug msg="Serving default certificate for request: \"\""

Any further ideas on how debug this, @ldez?

ldez · November 3, 2019, 1:10am

The log is not a problem (I got the same).

Could you try this:

version: '3.7'
services:
  web:
    image: traefik:2.0.4
    command:
      - "--entryPoints.https.address=:443"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--log.level=DEBUG"
    ports:
      - 443:443
      - 8080:8080
    depends_on:
      - backend
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro

  backend:
    image: containous/whoami
    command:
      - "--port=5678"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.backend.rule=HostRegexp(`{host:.+}`)"
      - "traefik.http.routers.backend.entrypoints=https"
      - "traefik.http.routers.backend.tls=true"
      - "traefik.http.services.backend.loadbalancer.server.port=5678"

and call the API:

$ curl -s http://127.0.0.1:8080/api/rawdata | jq

{
  "routers": {
    "backend@docker": {
      "entryPoints": [
        "https"
      ],
      "service": "backend",
      "rule": "HostRegexp(`{host:.+}`)",
      "tls": {},
      "status": "enabled",
      "using": [
        "https"
      ]
    }
  },
  "services": {
    "backend@docker": {
      "loadBalancer": {
        "servers": [
          {
            "url": "http://172.23.0.2:5678"
          }
        ],
        "passHostHeader": true
      },
      "status": "enabled",
      "usedBy": [
        "backend@docker"
      ],
      "serverStatus": {
        "http://172.23.0.2:5678": "UP"
      }
    }
  }
}

tcurdt · November 3, 2019, 1:25am

Just to be sure I updated docker to the same version:

Docker version 19.03.4, build 9013bf5

Then started your latest docker compose file

$ curl -s http://127.0.0.1:8080/api/rawdata | jq
{
  "routers": {
    "backend@docker": {
      "entryPoints": [
        "https"
      ],
      "service": "backend",
      "rule": "HostRegexp(`{host:.+}`)",
      "tls": {},
      "status": "enabled",
      "using": [
        "https"
      ]
    }
  },
  "services": {
    "backend@docker": {
      "loadBalancer": {
        "servers": [
          {
            "url": "http://172.23.0.2:5678"
          }
        ],
        "passHostHeader": true
      },
      "status": "enabled",
      "usedBy": [
        "backend@docker"
      ],
      "serverStatus": {
        "http://172.23.0.2:5678": "UP"
      }
    }
  }
}

And then did the test - and it is working for me as well.

$ curl -k -H Host:localhost https://127.0.0.1
Hostname: 6b5e6c192b5f
IP: 127.0.0.1
IP: 172.23.0.2
RemoteAddr: 172.23.0.3:52438
GET / HTTP/1.1
Host: localhost
User-Agent: curl/7.54.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 172.23.0.1
X-Forwarded-Host: localhost
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: 059c9f8e8d3b
X-Real-Ip: 172.23.0.1

I then verified the previous docker compose file to be working, too.

While that's good I hate not knowing the why:

Maybe a faulty network connection between traefik and the backend container resulted in the 404? But that should have given a different error message, no?
Maybe just a docker restart would have fixed it, too?
Maybe the docker update fixed it?

How frustrating to not know.

tcurdt · November 3, 2019, 1:34am

Interesting - I am back to the 404. Investigating further.

ldez · November 3, 2019, 1:39am

I recommend to do:

$ docker-compose down
$ docker-compose up --remove-orphans

$ docker ps

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                                  NAMES
235cbded33e2        traefik:2.0.4       "/entrypoint.sh --en…"   27 minutes ago      Up 27 minutes       0.0.0.0:443->443/tcp, 80/tcp, 0.0.0.0:8080->8080/tcp   tem_web_1
cea44e30ae28        containous/whoami   "/whoami --port=5678"    About an hour ago   Up 27 minutes       80/tcp                                                 tem_backend_1

$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
227b4abb185b        bridge              bridge              local
9b62799d98ec        host                host                local
d3326558f216        none                null                local
729231fd556a        tem_default         bridge              local

tcurdt · November 3, 2019, 1:06pm

OK - so I think I was just too tired last night and forget to set the Host header. So I think the initial 404 was docker glitch and we cannot find the actual cause at this stage anymore.

That said - it would be super cool if traefik could (at least in DEBUG) tell when there is no match for the given host. And not just return a 404 without leaving any clue. That would be my request for improvement.