Trying to get certs for services on another host

Traefik Traefik v3 (latest)
, ,
1

I'm pretty new to Traefik to start with. Everything working as expected with Traefik and I'll get certs for services under the same Docker host, but as soon as I try to do this with my Proxmox host or Pihole (this are the two I've tried with) it doesn't work.

I've done more or less exactly like Techno Tim does but for me it's not working.

docker-compose.yml:

services:
  traefik:
    container_name: traefik
    image: traefik:v3.3.3
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - proxy
    ports:
      - 80:80
      - 443:443
    environment:
      CF_DNS_API_TOKEN_FILE: /run/secrets/cf_api_token
      TRAEFIK_DASHBOARD_CREDENTIALS: ${TRAEFIK_DASHBOARD_CREDENTIALS}
    secrets:
      - cf_api_token
    env_file: .env
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /docker/traefik/traefik.yml:/traefik.yml:ro
      - /docker/traefik/acme.json:/acme.json
      - /docker/traefik/logs:/var/log/traefik
      - /docker/traefik/config.yml:/config.yml:ro
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`td.domain`)"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=${TRAEFIK_DASHBOARD_CREDENTIALS}"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`td.domain`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=cloudflare"
      - "traefik.http.routers.traefik-secure.tls.domains[0].main=domain"
      - "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.domain"
      - "traefik.http.routers.traefik-secure.service=api@internal"

secrets:
  cf_api_token:
    file: ./cf_api_token.txt

networks:
  proxy:
    name: proxy
    external: true

traefik.yml:

global:
  sendAnonymousUsage: false
api:
  dashboard: true
  debug: true
entryPoints:
  http:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https
  https:
    address: ":443"
serversTransport:
  insecureSkipVerify: true
providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
  file:
    filename: /config.yml
certificatesResolvers:
  cloudflare:
    acme:
      email: email@domain
      storage: acme.json
      caServer: https://acme-v02.api.letsencrypt.org/directory # prod (default)
        # caServer: https://acme-staging-v02.api.letsencrypt.org/directory # staging
      dnsChallenge:
        provider: cloudflare
        #disablePropagationCheck: true # uncomment this if you have issues pulling certificates through cloudflare, By setting this flag to true disables the need to wait for the propagation of the TXT record to all authoritative name servers.
        #delayBeforeCheck: 60s # uncomment along with disablePropagationCheck if needed to ensure the TXT record is ready before verification is attempted
        resolvers:
          - "1.1.1.1:53"
          - "1.0.0.1:53"

log:
  level: DEBUG

config.yml:

http:
 #region routers
  routers:
    proxmox:
      entryPoints:
        - "https"
      rule: "Host(`pve.domain`)"
      middlewares:
        - default-headers
        - https-redirectscheme
      tls: {}
      service: proxmox
    pihole:
      entryPoints:
        - "https"
      rule: "Host(`pihole.domain`)"
      middlewares:
        - default-headers
        - https-redirectscheme
      tls: {}
      service: pihole

#endregion
#region services
  services:
    proxmox:
      loadBalancer:
        servers:
          - url: "https://192.168.20.10:8006"
        passHostHeader: true
    pihole:
      loadBalancer:
        servers:
          - url: "http://192.168.1.10:80"
        passHostHeader: true

#endregion
  middlewares:

    https-redirectscheme:
      redirectScheme:
        scheme: https
        permanent: true

    default-headers:
      headers:
        frameDeny: true
        sslRedirect: true
        browserXssFilter: true
        contentTypeNosniff: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 15552000
        customFrameOptionsValue: SAMEORIGIN
        customRequestHeaders:
          X-Forwarded-Proto: https

    default-whitelist:
      ipWhiteList:
        sourceRange:
        - "192.168.0.0/16"
        - "172.16.0.0/12"
        - "10.0.0.0/8"

    secured:
      chain:
        middlewares:
        - default-whitelist
        - default-headers

My Proxmox host is on the same VLAN and there is no problems with the traffic between the VLANs either. In the logs I can't see any errors and to me it looks like it should work.

I can see this in the log if it says anything:

DBG github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:237 > Adding route for pihole.domain with TLS options default entryPointName=https

DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:29 > Creating middleware entryPointName=http middlewareName=redirect-http-to-https@internal middlewareType=RedirectScheme routerName=http-to-https@internal
DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:30 > Setting up redirection to https 443 entryPointName=http middlewareName=redirect-http-to-https@internal middlewareType=RedirectScheme routerName=http-to-https@internal
DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=http middlewareName=traefik-internal-recovery middlewareType=Recovery

DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:313 > Creating load-balancer entryPointName=https routerName=pihole@file serviceName=pihole@file
DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:350 > Creating server entryPointName=https routerName=pihole@file serverName=f3c24xxxx573dc serviceName=pihole@file target=http://192.168.1.10:80
DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:29 > Creating middleware entryPointName=https middlewareName=https-redirectscheme@file middlewareType=RedirectScheme routerName=pihole@file
DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:30 > Setting up redirection to https  entryPointName=https middlewareName=https-redirectscheme@file middlewareType=RedirectScheme routerName=pihole@file
DBG github.com/traefik/traefik/v3/pkg/middlewares/headers/headers.go:27 > Creating middleware entryPointName=https middlewareName=default-headers@file middlewareType=Headers routerName=pihole@file
DBG github.com/traefik/traefik/v3/pkg/middlewares/headers/headers.go:41 > Setting up secureHeaders from {map[X-Forwarded-Proto:https] map[] false [] [] [] [] [] 0 false [] [] map[] 15552000 true true true true SAMEORIGIN true true       false <nil> 0xc00141c9b0 <nil> <nil> <nil>} entryPointName=https middlewareName=default-headers@file middlewareType=Headers routerName=pihole@file
DBG github.com/traefik/traefik/v3/pkg/middlewares/headers/headers.go:47 > Setting up customHeaders/Cors from {map[X-Forwarded-Proto:https] map[] false [] [] [] [] [] 0 false [] [] map[] 15552000 true true true true SAMEORIGIN true true       false <nil> 0xc00141c9b0 <nil> <nil> <nil>} entryPointName=https middlewareName=default-headers@file middlewareType=Headers routerName=pihole@file
DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=https middlewareName=default-headers@file routerName=pihole@file

I've spent too many hours now trying to solve this without any luck.

2

If you enable TLS tls: {}, then you need to provide a TLS cert. If you want a LetsEncrypt TLS cert, then you should assign a certResolver to entrypoint or router.

In general I recommend a clean-up, place redirect and TLS globally on entrypoint, remove useless headers. Compare to simple Traefik example.

3

Is tls: {} necessary? I don't understand what I should provide. I have the acme.json file. Gone throught the docs and a lot of searching on the web, still don't understand what I have missed.

I just want to use my own domain with a trusted certificate to get rid of all the warnings. This is only on my local network.

4

You need to assign the certResolver. Simply check the configuration examples (doc):

## Dynamic configuration
http:
  routers:
    blog:
      rule: "Host(`example.com`)
      tls:
        certResolver: myresolver
5

That I have tried and it's the same issue still

6

It should give you a LetsEncrypt cert. What’s not working for you?

7

As I said in the initial post, I get certificates and all services under the Docker host itself working fine, but not for services on other hosts as Proxmox and Pi Hole (both running on their own machines).

8

Yes. You don’t get a LetsEncrypt TLS cert when you don't assign a certResolver to the router with rule and domain.

When just using tls: {}, you need to supply your own cert in a dynamic config file.

For LetsEncrypt to work, you need to assign the certResolver. Like in the example above and like in the linked repo.

If it’s still not working, make sure to supply new debug output.

9

With certResolver

   pihole:
      entryPoints:
        - "https"
      rule: "Host(`pihole.domain`)"
      tls:
        certResolver: cloudflare
      middlewares:
        - default-headers
        - https-redirectscheme
      service: pihole

DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [*.domain] The server validated our request lib=lego
DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [domain] acme: Trying to solve DNS-01 lib=lego
DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [domain] acme: Checking DNS record propagation. [nameservers=1.1.1.1:53,1.0.0.1:53] lib=lego
DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [pihole.domain] The server validated our request lib=lego
DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [pihole.domain] acme: Cleaning DNS-01 challenge lib=lego
DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [pihole.domain] acme: Validations succeeded; requesting certificates lib=lego
DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] Wait for propagation [timeout: 2m0s, interval: 2s] lib=lego
DBG github.com/go-acme/lego/v4@v4.21.0/log/logger.go:48 > [INFO] [pihole.domain] Server responded with a certificate. lib=lego
DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:730 > Certificates obtained for domains [pihole.domain] ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=cloudflare.acme routerName=pihole@file rule=Host(`pihole.domain`)
DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{},"tcp":{},"tls":{},"udp":{}} providerName=cloudflare.acme
DBG github.com/traefik/traefik/v3/pkg/tls/certificate.go:132 > Adding certificate for domain(s) pihole.domain
DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
[...]
DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:313 > Creating load-balancer entryPointName=https routerName=pihole@file serviceName=pihole@file
DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:350 > Creating server entryPointName=https routerName=pihole@file serverName=f3c24xxxx73dc serviceName=pihole@file target=http://192.168.1.10:80
DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:29 > Creating middleware entryPointName=https middlewareName=https-redirectscheme@file middlewareType=RedirectScheme routerName=pihole@file
DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:30 > Setting up redirection to https  entryPointName=https middlewareName=https-redirectscheme@file middlewareType=RedirectScheme routerName=pihole@file
DBG github.com/traefik/traefik/v3/pkg/middlewares/headers/headers.go:27 > Creating middleware entryPointName=https middlewareName=default-headers@file middlewareType=Headers routerName=pihole@file
DBG github.com/traefik/traefik/v3/pkg/middlewares/headers/headers.go:41 > Setting up secureHeaders from {map[X-Forwarded-Proto:https] map[] false [] [] [] [] [] 0 false [] [] map[] 15552000 true true true true SAMEORIGIN true true       false <nil> <nil> <nil> <nil> <nil>} entryPointName=https middlewareName=default-headers@file middlewareType=Headers routerName=pihole@file
DBG github.com/traefik/traefik/v3/pkg/middlewares/headers/headers.go:47 > Setting up customHeaders/Cors from {map[X-Forwarded-Proto:https] map[] false [] [] [] [] [] 0 false [] [] map[] 15552000 true true true true SAMEORIGIN true true       false <nil> <nil> <nil> <nil> <nil>} entryPointName=https middlewareName=default-headers@file middlewareType=Headers routerName=pihole@file
DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=https middlewareName=default-headers@file routerName=pihole@file
DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=https middlewareName=traefik-internal-recovery middlewareType=Recovery
DBG github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:237 > Adding route for pihole.domain with TLS options default entryPointName=https
[...]
DBG github.com/traefik/traefik/v3/pkg/tls/certificate.go:132 > Adding certificate for domain(s) pihole.domain
DBG github.com/traefik/traefik/v3/pkg/tls/certificate.go:132 > Adding certificate for domain(s) *.domain,domain
DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default

Does this say anything? With my non-existing experience it looks good though. In my browser on the Pihole web UI I only see the Pihole default cert.

Edit: In the Traefik Dashboard looking under the Pihole entry under HTTP Routers I can see that TLS is true and certificate resolver is set to cloudflare

I have successfully done this with Proxmox and Home Assistant now, for some reason Pi Hole refuse to work. Only getting a Error code: [SEC_ERROR_UNKNOWN_ISSUER] page.

10

You access Pi-Hole via pihole.domain?

11

When getting that error, yes.

12

Where do you get that? In Traefik debug log, in pihole log, in client/browser?

What does Traefik access log in JSON format (doc) tell you during request?

13

I get that in the browser.

The logs is empty. I get some output in the log file when accessing Proxmox for example, but nothing with Pihole.

For some reason it seems that the Pihole itself doesn't respect my DNS settings for it. One obvious thing i missed was that when pinging Pihole I see the IP of the Pihole instead of my Docker host as the other services. Piholes DNS is set to the Docker host though. I guess this issue have nothing to do with Traefik in that case?

Edit: I tried ping -A pihole.domain which returned the IP of the Pihole BUT my Docker host got rate limited with that command, which seems weird in this situation.

OMG, this was very embarrassing but it looks like it was an issue with Pihole where I'd changed DNS domain settings to my domain, but it should be default pi.hole. Thanks for pushing me in the right direction!

14

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.