Hey folks, i'm struggelig with this for hours, maybe someone can help me out:
I'm putting mailu (Mail framework for docker) behind traefik. Mailu will use letsencrypt to generate the certificates for itself and thus needs http-access to ./well-known and also a TCP router to the main TLS secured webservice.
My current setup is this.
traefik.yml:
api:
dashboard: true # Enable the dashboard
log:
filePath: "/logs/traefik.log"
level: INFO
certificatesResolvers:
letsEncrypt:
acme:
email: "letsEncrypt@maildomain" # Email address used for registration
storage: "/etc/traefik/acme/acme.json" # File or key used for certificates storage
caServer: https://acme-v02.api.letsencrypt.org/directory # prod (default)
tlsChallenge: {}
entryPoints:
http:
address: ":80" # Create the HTTP entrypoint on port 80
http:
redirections: # HTTPS redirection (80 to 443)
entryPoint:
to: "https" # The target element
scheme: "https" # The redirection target scheme
priority: 10000 # The Priority for the generated router
https:
address: ":443" # Create the HTTPS entrypoint on port 443
global:
checknewversion: true # Periodically check if a new version has been released.
sendanonymoususage: true # Periodically send anonymous usage statistics.
providers:
docker:
endpoint: "unix:///var/run/docker.sock" # Listen to the UNIX Docker socket
exposedByDefault: false # Only expose container that are explicitly enabled (using label traefik.enabled)
network: "traefik-net" # Default network to use for connections to all containers.
watch: true # Watch Docker Swarm events
file:
filename: "/etc/traefik/config.yml" # Link to the dynamic configuration
watch: true # Watch for modifications
providersThrottleDuration: 10 # Configuration reload frequency
config.yaml:
tcp:
middlewares:
internal:
ipAllowList:
sourceRange:
- "127.0.0.1/8"
- "192.168.1.7"
- "10.0.0.0/8"
http:
middlewares:
# A basic authentification middleware, to protect the Traefik dashboard to anyone except myself
# Use with traefik.http.routers.myRouter.middlewares: "traefikAuth@file"
traefikAuth:
basicAuth:
users:
- "admin:$apr1$ccwG8qS8$bAyTry5OU00VbG2XAgjBG0"
internal:
ipAllowList:
sourceRange:
- "127.0.0.1/8"
- "192.168.1.7"
- "10.0.0.0/8"
ratelimit:
rateLimit:
average: 100
burst: 200
# Recommended default middleware for most of the services
# Use with traefik.http.routers.myRouter.middlewares: "default@file"
# Equivalent of traefik.http.routers.myRouter.middlewares: "default-security-headers@file,error-pages@file,gzip@file"
default:
chain:
middlewares:
- default-security-headers
# - error-pages
- gzip
# Add automatically some security headers
# Use with traefik.http.routers.myRouter.middlewares: "default-security-headers@file"
default-security-headers:
headers:
browserXssFilter: true # X-XSS-Protection=1; mode=block
contentTypeNosniff: true # X-Content-Type-Options=nosniff
forceSTSHeader: true # Add the Strict-Transport-Security header even when the connection is HTTP
frameDeny: true # X-Frame-Options=deny
referrerPolicy: "strict-origin-when-cross-origin"
sslRedirect: true # Allow only https requests
stsIncludeSubdomains: true # Add includeSubdomains to the Strict-Transport-Security header
stsPreload: true # Add preload flag appended to the Strict-Transport-Security header
stsSeconds: 63072000 # Set the max-age of the Strict-Transport-Security header (63072000 = 2 years)
# Enables the GZIP compression (https://docs.traefik.io/middlewares/compress/)
# if the response body is larger than 1400 bytes
# if the Accept-Encoding request header contains gzip
# if the response is not already compressed (Content-Encoding is not set)
# Use with traefik.http.routers.myRouter.middlewares: "gzip@file"
gzip:
compress: {}
# See https://doc.traefik.io/traefik/https/tls/
tls:
options:
# To use with the label "traefik.http.routers.myrouter.tls.options=modern@file"
modern:
minVersion: "VersionTLS13" # Minimum TLS Version
sniStrict: true # Strict SNI Checking
# To use with the label "traefik.http.routers.myrouter.tls.options=intermediate@file"
intermediate:
cipherSuites:
- "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"
- "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"
- "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"
- "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"
- "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305"
- "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"
minVersion: "VersionTLS12" # Minimum TLS Version
sniStrict: true # Strict SNI Checking
# To use with the label "traefik.http.routers.myrouter.tls.options=old@file"
old:
cipherSuites:
- "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"
- "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"
- "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"
- "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"
- "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305"
- "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"
- "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"
- "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256"
- "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA"
- "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA"
- "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA"
- "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA"
- "TLS_RSA_WITH_AES_128_GCM_SHA256"
- "TLS_RSA_WITH_AES_256_GCM_SHA384"
- "TLS_RSA_WITH_AES_128_CBC_SHA256"
- "TLS_RSA_WITH_AES_128_CBC_SHA"
- "TLS_RSA_WITH_AES_256_CBC_SHA"
- "TLS_RSA_WITH_3DES_EDE_CBC_SHA"
minVersion: "TLSv1" # Minimum TLS Version
sniStrict: true # Strict SNI Checking
docker-compose.yaml (only the container in question as it is long):
# This file is auto-generated by the Mailu configuration wizard.
# Please read the documentation before attempting any change.
# Generated for compose flavor
services:
# External dependencies
redis:
image: redis:alpine
restart: always
volumes:
- "/mailu/redis:/data"
depends_on:
- resolver
dns:
- 192.168.203.254
[...]
# Core services
front:
image: ${DOCKER_ORG:-ghcr.io/mailu}/${DOCKER_PREFIX:-}nginx:${MAILU_VERSION:-2024.06}
restart: always
env_file: stack.env
logging:
driver: journald
options:
tag: mailu-front
# ports:
# - "127.0.0.1:80:80"
# - "127.0.0.1:443:443"
# - "127.0.0.1:25:25"
# - "127.0.0.1:465:465"
# - "127.0.0.1:587:587"
# - "127.0.0.1:110:110"
# - "127.0.0.1:995:995"
# - "127.0.0.1:143:143"
# - "127.0.0.1:993:993"
# - "127.0.0.1:4190:4190"
networks:
- default
- traefik-net
volumes:
- "/mailu/certs:/certs"
- "/mailu/overrides/nginx:/overrides:ro"
depends_on:
- resolver
dns:
- 192.168.203.254
labels:
traefik.enable: "true"
# second part is important to ensure Mailu can get all certificates from letsencrypt for all the hostnames
traefik.http.routers.mailuweb.rule: "(Host(`maildomain`) || HOST(`domain1`)) && PathPrefix(`/.well-known/acme-challenge/`)"
traefik.http.routers.mailuweb.entrypoints: "http"
traefik.http.routers.mailuweb.priority: 20000
traefik.http.services.mailuweb.loadbalancer.server.port: "80"
# other FQDNS can be added here:
traefik.tcp.routers.mailuhttps.rule: "HostSNI(`maildomain`)"
traefik.tcp.routers.mailuhttps.entrypoints: "https"
traefik.tcp.routers.mailuhttps.middlewares: "internal@file"
traefik.tcp.routers.mailuhttps.tls.passthrough: "true"
traefik.tcp.routers.mailuhttps.tls.options: "intermediate@file"
traefik.tcp.routers.mailuhttps.service: "mailuadmin"
traefik.tcp.services.mailuadmin.loadbalancer.server.port: "443"
traefik.tcp.services.mailuadmin.loadbalancer.proxyProtocol.version: "2"
# traefik.tcp.routers.smtp.rule: "HostSNI(`*`)"
# traefik.tcp.routers.smtp.entrypoints: "smtp"
# traefik.tcp.routers.smtp.service: "smtp"
# traefik.tcp.services.smtp.loadbalancer.server.port: "25"
# traefik.tcp.services.smtp.loadbalancer.proxyProtocol.version: "2"
# traefik.tcp.routers.submissions.rule: "HostSNI(`*`)"
# traefik.tcp.routers.submissions.entrypoints: "submissions"
# traefik.tcp.routers.submissions.service: "submissions"
# traefik.tcp.services.submissions.loadbalancer.server.port: "465"
# traefik.tcp.services.submissions.loadbalancer.proxyProtocol.version: "2"
# traefik.tcp.routers.imaps.rule: "HostSNI(`*`)"
# traefik.tcp.routers.imaps.entrypoints: "imaps"
# traefik.tcp.routers.imaps.service: "imaps"
# traefik.tcp.services.imaps.loadbalancer.server.port: "993"
# traefik.tcp.services.imaps.loadbalancer.proxyProtocol.version: "2"
healthcheck:
test: ['NONE']
[...]
The network traefik-net is generated externally (in portainer), traefik itself is deployed via docker in portainer as well.
The problem is: The container can get a certificate via acme-challenge but when I try to connect to it, Firefox says: PR_END_OF_FILE_ERROR or SSL_ERROR_RX_RECORD_TOO_LONG.
In curl it looks like this:
curl -vv https://maildomain/
* Host maildomain:443 was resolved.
* IPv6: 2003:xxxxxxxx
* IPv4: 124.xxxx
* Trying [2003:xxxxxxxx]:443...
* Connected to maildomain (2003:xxxxxxxx) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* OpenSSL/3.0.13: error:0A00010B:SSL routines::wrong version number
* Closing connection
curl: (35) OpenSSL/3.0.13: error:0A00010B:SSL routines::wrong version number
I checked from inside the actual traefik-docker and it seems, traefik can connect to docker with no problems at all:
/ # wget --no-check-certificate https://172.18.0.3
Connecting to 172.18.0.3 (172.18.0.3:443)
Connecting to 172.18.0.3 (172.18.0.3:443)
saving to 'index.html'
index.html 100% |********************************************************************************************************************************************| 7151 0:00:00 ETA
'index.html' saved
Mailu itself gives this output:
172.18.0.2 - - [25/Apr/2025:19:27:14 +0000] "\x00" 400 150 "-" "-"
172.18.0.2 - - [25/Apr/2025:19:27:14 +0000] "\x00" 400 150 "-" "-"
172.18.0.2 - - [25/Apr/2025:19:28:05 +0000] "\x00" 400 150 "-" "-"
172.18.0.2 - - [25/Apr/2025:19:28:05 +0000] "\x00" 400 150 "-" "-"
172.18.0.2 - - [25/Apr/2025:19:29:00 +0000] "\x00" 400 150 "-" "-"
172.18.0.2 - - [25/Apr/2025:19:29:00 +0000] "\x00" 400 150 "-" "-"
I've fiddled around with the configs a lot but cannot seem to find any hint on why whis fails. When putting traefik into debug-mode i get this:
2025-04-26T09:15:31+02:00 DBG github.com/traefik/traefik/v3/pkg/tcp/proxy.go:41 > Handling TCP connection address=172.18.0.3:443 remoteAddr=10.0.99.10:35978
2025-04-26T09:15:31+02:00 DBG github.com/traefik/traefik/v3/pkg/tcp/proxy.go:104 > Error while terminating TCP connection error="close tcp 172.18.0.2:55184->172.18.0.3:443: use of closed network connection"
2025-04-26T09:15:31+02:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/tcp/ipallowlist/ip_allowlist.go:60 > Connection from 10.0.99.10:35990 accepted middlewareName=internal@file middlewareType=IPAllowListerTCP
2025-04-26T09:15:31+02:00 DBG github.com/traefik/traefik/v3/pkg/tcp/proxy.go:41 > Handling TCP connection address=172.18.0.3:443 remoteAddr=10.0.99.10:35990
2025-04-26T09:15:31+02:00 DBG github.com/traefik/traefik/v3/pkg/tcp/proxy.go:104 > Error while terminating TCP connection error="close tcp 172.18.0.2:55192->172.18.0.3:443: use of closed network connection"
That is really odd, as the connection is working (like i checked from the container). Can anyone help me out on why this is failing?