DNS-over-TLS almost working

Traefik Traefik v2 (latest)
,
1

Hi!

I'm trying to implement DNS-over-TLS using pihole behind Traefik.

My setup is almost working, but despite Traefik not setup to passthrough the TLS to the TCP service the service is receiving encrypted data. I found that out doing a tcpdump of the port 53 on my pihole docker interface.

If I do a dig @pihole_docker_ip_address linux.org I can see in the tcpdump that the packets are in clear text and I get a result to my query.

When I try to connect my Android client to my DoT setup Traefik throws me an error

"Error during connection: readfrom tcp 172.18.0.2:49132->172.18.0.4:53: remote error: tls: expired certificate"

172.18.0.2 being Traefik and 172.18.0.4 being the pihole DNS service.
tcpdump show encrypted packets coming from Traefik to the dns interface.

All my certs are valid and working, I'm using CloudFlare API.
Traefik WebUI show that my router is not in passthrough but yet traffic seems to be passthrough encrypted. How can I force Traefik to terminate the TLS connection and pass the packets decrypted to my service?

SjVsNWK1
SjVsNWK11427×842 50.3 KB

2

Thanks for your interest in Traefik!

The documentation has more info about TLS termination.

See this YAML example:

## Dynamic configuration
tcp:
  routers:
    Router-1:
      rule: "HostSNI(`foo-domain`)"
      service: service-id
      # will terminate the TLS request by default
      tls: {}
3

Thank you for your reply.

I'm using labels in docker compose and my router does need to serve a tls certificate. Here what I have regarding this router at the moment in my docker compose.

services:
  pihole:
    ...
    labels:
      ...
      # DNS-over-TLS
      - "traefik.tcp.routers.pihole-dot.rule=HostSNI(`foo-domain`)"
      - "traefik.tcp.routers.pihole-dot.entrypoints=dnsovertls"
      - "traefik.tcp.routers.pihole-dot.tls.certresolver=letsencrypt"
      - "traefik.tcp.routers.pihole-dot.service=pihole-tcp"
      - "traefik.tcp.services.pihole-tcp.loadbalancer.server.port=53"
      ...

Entrypoint dnsovertls is defined as follow in my traefik config file:

entryPoints:
  dnsovertls:
    address: ":853"
4

Can you share your full Traefik static and dynamic config, and docker-compose.yml if used?

Enable Traefik debug log and check for "error".

5

Here are my configs! Thanks again for the interest you have in my problem, I'm sure a lot of guys would like to be able to do the same as me.

docker-compose.yml for Traefik 
version: "3.8"

services:
  traefik:
    image: "traefik:latest"
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - "no-new-privileges:true"
    networks:
      - proxy
    ports:
      - "80:80"
      - "443:443"
      - "853:853"
    volumes:
      - "/etc/localtime:/etc/localtime:ro"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./traefik-data/traefik.yml:/traefik.yml:ro"
      - "./traefik-data/acme.json:/acme.json"
      - "./traefik-data/configurations:/configurations"
    environment:
      - "CF_DNS_API_TOKEN=<cf_token>"
    labels:
      - traefik.enable=true
      - traefik.docker.network=proxy
      - traefik.http.routers.traefik-secure.entrypoints=websecure
      - traefik.http.routers.traefik-secure.rule=Host(`traefik.example.com`)
      - traefik.http.routers.traefik-secure.service=api@internal
      - traefik.http.routers.traefik-secure.middlewares=user-auth@file
traefik.yml 
api:
  dashboard: true

log:
  level: INFO
  format: common

#accesslog:
#  format: common

entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websecure
  websecure:
    address: ":443"
    http:
      middlewares:
        - secureHeaders@file
      tls:
        certResolver: letsencrypt
  dnsovertls:
    address: ":853"

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
  file:
    filename: /configurations/dynamic.yml

certificatesResolvers:
  letsencrypt:
    acme:
      email: me@privacy.net
      storage: acme.json
      dnschallenge:
        provider: cloudflare
dynamic.yml 
# Dynamic configuration
http:
  middlewares:
    secureHeaders:
      headers:
        sslRedirect: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 31536000
    user-auth:
      basicAuth:
        users:
          - "me:password"

tls:
  stores:
    default:
      defaultGeneratedCert:
        resolver: letsencrypt
        domain:
          main: example.com
          sans:
            - dns.example.com
  options:
    default:
      cipherSuites:
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
      minVersion: VersionTLS12
docker-compose.yml for pihole 
version: "3.9"

networks:
  proxy:
    external: true

services:
  pihole:
    image: pihole/pihole:latest
    container_name: pihole
    hostname: dns.example.com
    networks:
      - proxy
    environment:
      PUID: '1001'
      PGID: '1001'
      TZ: 'America/Toronto'
      WEBPASSWORD: 'best_strong_password'
    volumes:
      - './etc-pihole/:/etc/pihole/'
      - './etc-dnsmasq.d/:/etc/dnsmasq.d/'
    dns:
      - 1.0.0.1
      - 1.1.1.1
    expose:
      - 80
      - 53
      - 53/udp
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=proxy"

      # web interface
      - "traefik.http.routers.pihole-gui.rule=Host(`dns.example.com`)"
      - "traefik.http.routers.pihole-gui.entrypoints=websecure"
      - "traefik.http.routers.pihole-gui.middlewares=pihole_PiholeChain"
      - "traefik.http.services.pihole-gui.loadbalancer.server.port=80"

      # make sure '/admin' is there
      - "traefik.http.middlewares.pihole_AddAdminPath.replacepathregex.regex=^/((?i:(admin)/{0,1}|.{0})(.*))"
      - "traefik.http.middlewares.pihole_AddAdminPath.replacepathregex.replacement=/admin/$$3"
      - "traefik.http.middlewares.pihole_PiholeChain.chain.middlewares=pihole_AddAdminPath,secureHeaders@file"

      # DNS-over-TLS
      - "traefik.tcp.routers.pihole-dot.rule=HostSNI(`dns.example.com`)"
      - "traefik.tcp.routers.pihole-dot.entrypoints=dnsovertls"
      - "traefik.tcp.routers.pihole-dot.tls.certresolver=letsencrypt"
      - "traefik.tcp.routers.pihole-dot.tls=true"
      - "traefik.tcp.routers.pihole-dot.service=pihole-tcp"
      - "traefik.tcp.services.pihole-tcp.loadbalancer.server.port=53"

The error I'm getting is this one:
traefik | time="2023-01-18T03:24:32Z" level=error msg="Error during connection: readfrom tcp 172.18.0.2:38970->172.18.0.4:53: remote error: tls: expired certificate"

It appear when I try to setup DoT on my Android as a client. I get no other error than that one... Very non descriptive.

6

Did you manage to solve it?

I have it working and these are my labels for DoT for the Adguard container:

# DNS-over-TLS
  - "traefik.tcp.routers.dot.rule=HostSNI(`adguard.example.com`)"
  - "traefik.tcp.routers.dot.entrypoints=dot"
  - "traefik.tcp.routers.dot.tls.passthrough=true"
  - "traefik.tcp.routers.dot.service=dot"
  - "traefik.tcp.services.dot.loadbalancer.server.port=853"

I also had to add this to my TLS options:

alpnProtocols:
- http/1.1
- h2
- acme-tls/1
- dot

1 Like
7

Nope it still doesn't work. I haven't worked on it since though.

I tried to add the alpnProtocols like you suggest but it didn'e help.

AdGuard does handle DoT by itself this is why you can connect to your service using port 853.

Pi-Hole does not implement DoT in itself.

The error I'm getting now is :

traefik | time="2023-02-02T15:31:53Z" level=error msg="Error during connection: readfrom tcp 172.18.0.2:50072->172.18.0.4:53: remote error: tls: expired certificate"

Traefik being .2 and pi-hole geing .4

Look like Traefik still want to establish a secure connection to pihole, but it cannot port 53 does not supply any certificate, heck pihole doesn't do TLS.

I just validated from extracting information from acme.json that my certificate for that HostSNI is valid and not expired.

8

Am I the only one trying to make pi-hole work in DNS-over-TLS behind Traefik?

9

I am running into the same issue with android 10 and lower. I could be that the older android version uses the older lets encrypt root certificate? (Android devices with DoT configured; interaction with new default chain - #14 by jsuelwald - Help - Let's Encrypt Community Support)

Edit:
Indeed it has to do with the root cert.
I fixed it by adding:

preferredChain: 'ISRG Root X1'

So traefik.yml becomes:

certificatesResolvers:
  lets-encrypt:
    acme:
      #caserver: https://acme-staging-v02.api.letsencrypt.org/directory #only for debug
      email: {{emailaddress}}
      storage: /letsencrypt/acme.json
      tlschallenge: true
      preferredChain: 'ISRG Root X1'

Remove your current acme.json file and restart traefik.

10

Thanks for sharing your solution. I went an other direction with this problem in the end. I ended up using "Technitium DNS Server" which support DNS-over-TLS directly, so I don't have to fiddle with Traefik. The only thing with this solution tough is that I have to have a ceperate certbot running for the certificate of my DoT server, because the generated certificate need to be transformed to PKCS #12 for Technitium can use it. It's not been 90 days yet, so I expect something breaking when the cert renew, I'll adjust the convert script to restart the container if needed. Also I find "Technitium DNS Server" a little more geek like than pi-hole.