Do not route tls passthrough to backend service with docker provider

Traefik Traefik v2
1

Hi, I setup a service (fleet) which using it's certificates to export https service, and I tried to configure below lables:

    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=traefik_default"
      - "traefik.tcp.services.fleetdm-https.loadbalancer.server.port=8080"
      - "traefik.tcp.routers.fleetdm-https.rule=HostSNI(`fleet.home.mengz.lan`)"
      - "traefik.tcp.routers.fleetdm-https.entrypoints=websecure"
      - "traefik.tcp.routers.fleetdm-https.tls=true"
      - "traefik.tcp.routers.fleetdm-https.tls.passthrough=true"

And see it in traefik dashboard

Screenshot_20230201_174327
Screenshot_20230201_1743271153×893 49.4 KB

When I access https://fleet.home.mengz.lan/, it always repot "404 page not found".
But when I docker exec into traefik, and execute

/ # curl -k https://172.20.0.15:8080/
<!doctype html><html><head><meta charset="UTF-8"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="robots" content="noindex"/><link rel="stylesheet" href="/assets/bundle-3058a80299be2c449263.css"/><link rel="shortcut icon" href="/assets/favicon.ico"/><title>Fleet for osquery</title><script>var urlPrefix = "";</script></head><body><div id="app"></div><script async defer="defer" src="/assets/bundle-afe3a92ee33eddc0def7.js" onload="this.parentElement.removeChild(this)"></script><script>document.addEventListener("touchstart", function () {}, false);</script></body></html>/

Can get the response. If I directly expose the port 8080 to host, it's also ok.

Is there some wrong configurations?

2

If your target service handles all TLS, then you don't want to set any TLS in Traefik, as Traefik should just pass plain TCP stream directly to your target.

If you set tls=true, than Traefik will look for a certificate and create a default one if not found.

Not sure if it's a good idea to use a TCP router on websecure, if you use other HTTP services on the same entrypoint.

If it still does not work, show your full config, Traefik static and dynamic config, docker-compose.yml if used.

3

@bluepuma77 Thanks for your help!

Yes, the target backend service (fleet) is a https service (should handle all TLS), I want to use the same https entrypoint(443) for ingress, and I tried below configs

    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=traefik_default"
      - "traefik.http.services.fleetdm-https.loadbalancer.server.port=8080"
      - "traefik.http.services.fleetdm-https.loadbalancer.server.scheme=https"
      - "traefik.http.routers.fleetdm-https.rule=Host(`fleet.home.mengz.lan`)"
      - "traefik.http.routers.fleetdm-https.entrypoints=websecure"
      - "traefik.http.routers.fleetdm-https.service=fleetdm-https"
        #      - "traefik.tcp.routers.fleetdm-https.tls=true"
        #      - "traefik.tcp.routers.fleetdm-https.tls.passthrough=true"

the result is the same, the traefik did not proxy the request to backend service.

below is my traefik configuration:

    ports:
    - "80:80/tcp"
    - "443:443/tcp"
    volumes:
    - "/etc/localtime:/etc/localtime:ro"
    - "/var/run/docker.sock:/var/run/docker.sock"
    - "./confs/:/confs/"
    command:
    - "--entrypoints.web.address=:80"
    - "--entrypoints.websecure.address=:443"
    - "--api=true"
    - "--api.debug=true"
    - "--providers.docker"
    - "--providers.docker.exposedbydefault=false"
    - "--providers.docker.watch=true"
    - "--providers.file.directory=/confs/"
    - "--providers.file.watch=true"
    - "--log.level=ERROR"
    - "--accesslog=true"

and a configuration file in /confs/ :

tls:
  stores:
    default:
      defaultCertificate:
        certFile: /confs/ssl/cert.cert
        keyFile: /confs/ssl/cert.key
4

Are those certs the same TLS/SSL certs that are used by fleet?

5

Yes, the certificates used by fleet is same with defaultCertificate in Traefik.

6

If Traefik does have the certs available and the protocol is http, try switching the router (all labels) to http.

      …
      - "traefik.http.routers.fleetdm-https.tls=true"
      - "traefik.http.routers.fleetdm-https.tls.passthrough=true"

Try using .rule=Host().

Enable Traefik debug log to see what’s happening.

7

@bluepuma77 Please see below logs:

traefik-proxy  | time="2023-02-03T19:23:33+08:00" level=debug msg="Skipping unchanged configuration." providerName=docker
traefik-proxy  | time="2023-02-03T19:23:34+08:00" level=debug msg="Provider event received {Status:start ID:5b59cbb19cf7617ec982146b77c90c7635929adb16f612d80a2f5f8c08352ccc From:fleetdm/fleet:latest Type:container Action:start Actor:{ID:5b59cbb19cf7617ec982146b77c90c7635929adb16f612d80a2f5f8c08352ccc Attributes:map[com.docker.compose.config-hash:fb74b6d4c2b3453a0a5c803725c7bfdeac870b70f79ac61fc5ef782fa6f9bfb3 com.docker.compose.container-number:1 com.docker.compose.depends_on:oneshot:service_completed_successfully com.docker.compose.image:sha256:797bd78ce296db27602bdd0501224c95dda5f58e0d66966dedc260cf97efc1fb com.docker.compose.oneoff:False com.docker.compose.project:fdm com.docker.compose.project.config_files:/home/mengz/dockerapps/fdm/compose.yml com.docker.compose.project.working_dir:/home/mengz/dockerapps/fdm com.docker.compose.service:fleet com.docker.compose.version:2.15.1 image:fleetdm/fleet:latest maintainer:Fleet Developers name:fdm-fleet traefik.docker.network:traefik_default traefik.enable:true traefik.http.routers.fleetdm-https.entrypoints:websecure traefik.http.routers.fleetdm-https.rule:Host(`fleet.home.mengz.lan`) traefik.http.routers.fleetdm-https.service:fleetdm-https traefik.http.routers.fleetdm-https.tls:true traefik.http.routers.fleetdm-https.tls.passthrough:true traefik.http.services.fleetdm-https.loadbalancer.server.port:8080 traefik.http.services.fleetdm-https.loadbalancer.server.scheme:https]} Scope:local Time:1675423414 TimeNano:1675423414312898424}" providerName=docker
traefik-proxy  | time="2023-02-03T19:23:35+08:00" level=debug msg="Skipping unchanged configuration." providerName=docker
traefik-proxy  | time="2023-02-03T19:24:26+08:00" level=debug msg="Serving default certificate for request: \"fleet.home.mengz.lan\""
traefik-proxy  | 192.168.31.110 - - [03/Feb/2023:11:24:26 +0000] "GET / HTTP/2.0" 404 19 "-" "-" 1 "-" "-" 0ms
traefik-proxy  | 192.168.31.110 - - [03/Feb/2023:11:24:27 +0000] "GET /favicon.ico HTTP/2.0" 404 19 "-" "-" 2 "-" "-" 0ms

and can not see the http routers item on traefik dashboard.

    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=traefik_default"
      - "traefik.http.services.fleetdm-https.loadbalancer.server.port=8080"
      - "traefik.http.services.fleetdm-https.loadbalancer.server.scheme=https"
      - "traefik.http.routers.fleetdm-https.rule=Host(`fleet.home.mengz.lan`)"
      - "traefik.http.routers.fleetdm-https.entrypoints=websecure"
      - "traefik.http.routers.fleetdm-https.service=fleetdm-https"
      - "traefik.http.routers.fleetdm-https.tls=true"
      - "traefik.http.routers.fleetdm-https.tls.passthrough=true"
8

Check you logs: Is Traefik provider.file reading your dynamic config file? Is Traefik provider.docker recognizing your target service?

How do you deploy? Do you use Docker Swarm? Note that depending on deployment method labels need to be within service or within service.deploy in docker-compose.yml.

9

@bluepuma77 As the 2nd line logs I provided in my previous replay, end with "TimeNano:1675423414312898424}" providerName=docker", so I think it's read by provider.docker.

And I deploy use single docker engine (not swarm), so the labels within service.

10

Please check the Traefik log:

11

I grep the logs when I start the 'fleet' service，no items related to 'provider.file'.

12

I want to know if Traefik loads your dynamic config file successfully (for TLS certs).

13

Yes, I think so, because orther backend services are ok with traefik tls (use the default certs) termination.

14

hi @mengzyou Is it possible to know how you did it?
i also experience the same problem, i have traefik, which needs to contact fleet in ssl on 8080 but i get Service Unavailable
Thanks