Traefik to Traefik mTLS authentication

johnnyhuy · July 15, 2019, 8:04am

Hey!

So I've been trying to setup a localhost Traefik to server Traefik instance.

I'm trying to send a HTTP request to a localhost DNS to route to a whoami web server at the server backend to read headers. Both instances are running on Docker images with Docker Compose.

Localhost Traefik config

insecureSkipVerify = true

[entryPoints]
  [entryPoints.http]
  address = ":80"
  [entryPoints.https]
  address = ":443"
  [entryPoints.https.tls]
    [entryPoints.https.tls.ClientCA]
    files = ["/var/certs/cert.pem"]
    optional = true
    [[entryPoints.https.tls.certificates]]
    certFile = "/var/certs/cert.pem"
    keyFile = "/var/certs/key.pem"

[file]
[frontends]
  [frontends.frontend1]
  backend = "backend1"
    [frontends.frontend1.routes.test_1]
    rule = "Host:test.localhost"
  [frontends.frontend1.passTLSClientCert]
    pem = true

[backends]
  [backends.backyard]
    [backends.backyard.servers.server1]
    url = "https://example.com"
    weight = 10

Server Traefik config

insecureSkipVerify = true

[entryPoints]
  [entryPoints.http]
  address = ":80"

  [entryPoints.https]
  address = ":443"
  [entryPoints.https.tls]
    [entryPoints.https.tls.ClientCA]
    files = ["/certs/cert.pem"]
    optional = false
    [[entryPoints.https.tls.certificates]]
    certFile = "/certs/cert.pem"
    keyFile = "/certs/key.pem"

Here's a diagram, hopefully this is clear enough.

Every time I send a GET request to https://test.localhost I should get to https://example.com through the two Traefik instances. But I keep getting a 502 bad certificate error coming from the receiving Traefik instance. I suspect that the certificates aren't getting validated on the server end.

Is there a way this can be done with Traefik V1 or V2? Or if it's even possible?

jbd · July 15, 2019, 10:26am

Hi @johnnyhuy,

The following post should answer your question: mTLS - Add client cert on outbound

johnnyhuy · July 15, 2019, 10:26pm

Hey @jbd thanks for the quick response however, I've tried out V1 with the following settings you mentioned on GitHub but I can't seem to get it working pass the second Traefik instance from the same "Bad Gateway" 502 error. Am I missing something?

Keep in mind that the client doesn't supply the certificate but the Traefik instance itself forwards it to another Traefik instance.

Also, can this be replicated in V2 at its current state?

jbd · July 16, 2019, 7:35am

Can you provide your configuration files (toml && docker-compose.yml)?

Like I said, this is more a hack than a real solution in the Traefik v1.
So, we are working on a better solution for the v2 but I don't know when it will be integrated.

johnnyhuy · July 16, 2019, 7:58am

Sure, here's the following:

Localhost Docker Compose YAML

version: '3'

services:
  traefik:
    image: traefik:alpine
    command: --api --docker
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./traefik.toml:/etc/traefik/traefik.toml
      - ./log:/var/log
      # self-signed certs here
      - ./cert.pem:/var/certs/cert.pem
      - ./key.pem:/var/certs/key.pem

Localhost Traefik TOML

The same as mentioned in the original post.

insecureSkipVerify = true

[entryPoints]
  [entryPoints.http]
  address = ":80"
  [entryPoints.https]
  address = ":443"
  [entryPoints.https.tls]
    # This Traefik instance passes without a client certificate
    [entryPoints.https.tls.ClientCA]
    # Should I expect Traefik to forward this cert?
    files = ["/var/certs/cert.pem"]
    optional = true
    [[entryPoints.https.tls.certificates]]
    certFile = "/var/certs/cert.pem"
    keyFile = "/var/certs/key.pem"

[file]
[frontends]
  [frontends.frontend1]
  backend = "backend1"
    [frontends.frontend1.routes.test_1]
    rule = "Host:test.localhost"
  [frontends.frontend1.passTLSClientCert]
    pem = true

[backends]
  [backends.backyard]
    [backends.backyard.servers.server1]
    url = "https://example.com"
    weight = 10

Remote server Traefik TOML

insecureSkipVerify = true

[entryPoints]
  [entryPoints.traefik]
  address = ":8080"

  [entryPoints.http]
  address = ":80"

  [entryPoints.https]
  address = ":443"
  [entryPoints.https.tls]
    # This Traefik instance requires a client cert to pass
    [entryPoints.https.tls.ClientCA]
    files = ["/certs/cert.pem"]
    optional = false
    [[entryPoints.https.tls.certificates]]
    certFile = "/certs/cert.pem"
    keyFile = "/certs/key.pem"

[docker]
endpoint = "unix:///var/run/docker.sock"
domain = "docker.localhost"
watch = true
exposedByDefault = true

Remote server Docker Compose YAML

version: '3'

services:
  traefik:
    build:
      context: $PWD/destination/
      dockerfile: $PWD/destination/Dockerfile
    command: --api
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      # the same self-signed certs from the localhost Traefik instance
      - ./cert.pem:/var/certs/cert.pem
      - ./key.pem:/var/certs/key.pem
  whoami:
    image: containous/whoami
    labels:
      - "traefik.frontend=destination"
      - "traefik.backend=destination"
      - "traefik.frontend.entryPoints=http,https"
      - "traefik.frontend.passHostHeader=true"
      - "traefik.frontend.rule=Host:example.com"
      - "traefik.frontend.passTLSClientCert.infos.sans=true"
      - "traefik.frontend.passTLSClientCert.pem=true"

That's all good for V2, I'll keep an eye out for any major changes in that area

jbd · July 17, 2019, 9:09am

Hi @johnnyhuy,

Sorry for the delay,

the hack works with the deprecated passTLSCert option:
traefik.frontend.passTLSCert=true

The passTLSClientCert configuration allows to pass the client certificate to the backend in a specific header.

I didn't test your use case for now but I will try to do it as soon as possible.

johnnyhuy · July 17, 2019, 9:23am

Aha! Yes it did indeed work on with the deprecated passTLSCert as you mentioned.

The entire time I was using passTLSClientCert thinking that the deprecated feature was going to produce the same result. Is this intended?

Thanks again, feel free to test it whenever you can

jbd · July 17, 2019, 10:43am

I'm glad that solves your issue.

The passTLSCert option doesn't work as expected that's why she is deprecated and the passTLSClientCert exists.

johnnyhuy · July 17, 2019, 10:46pm

Hey @jbd I've swapped out passTLSCert with passTLSClientCert base on Traefik v1.7 Common - Pass TLS Client Cert and I don't seem to be getting the same result.

Correct me if I'm wrong, its because it's passed in a specific header?

[frontends]
  [frontends.frontend1]
  backend = "backyard"
  passHostHeader = true
  # passTLSCert = true
  [frontends.frontend1.passTLSClientCert]
    pem = true
  [frontends.frontend1.passTLSClientCert.infos]
    notBefore = true
    notAfter = true
  [frontends.frontend1.passTLSClientCert.infos.subject]
    country = true
    domainComponent = true
    province = true
    locality = true
    organization = true
    commonName = true
    serialNumber = true
  [frontends.frontend1.passTLSClientCert.infos.issuer]
    country = true
    domainComponent = true
    province = true
    locality = true
    organization = true
    commonName = true
    serialNumber = true
  [frontends.frontend1.routes.test_1]
  rule = "Host:test.localhost"

jbd · July 18, 2019, 6:54am

Using the deprecated option to have a mTLS between Traefik and a backend is a hack that I don't recommend.
The PassTLSClientCert works as expected, it extracts the client cert and pass it to the backend through a specific header and it doesn't applied any TLS connection.

johnnyhuy · July 22, 2019, 5:06am

Hello again

I've used PassTLSClientCert to try to pass the cert to the destination Traefik reverse proxy that has mTLS enabled with the optional field set to false.

This means the request must require the certificate to pass that reverse proxy. But I'm still getting a "bad certificate" error compared to using passTLSCert. What's the right way to do it? (without using deprecated features)

jbd · July 22, 2019, 7:08am

Hi @johnnyhuy,

Sadly, there is no way to do this, for now, without this passTLSCert hack.
This feature will come in the v2.

johnnyhuy · July 23, 2019, 3:43am

That's unfortunate , I've posted a feature request on GitHub in regards to the scenario.

jbd · December 9, 2020, 10:12am

Hi @johnnyhuy

A little update for the v2, the PR #7203 allows to tune connection per service thanks to serverstransport, so you will be able to configure mtls.

It will be shipped in the v2.4, and before you ask, the v2.4-rc1 will come in the coming weeks.