Mailu behind Traefik (docker)

Traefik Traefik v3 (latest)
1

I am definitely missing something in the way Traefik routes ports.
I can get it to work fine when there's only one port of interest.

PS - I couldn’t figure out how to break “links”, so I put a space between the slashes.

mail.env (relavent parts)

# Subnet of the docker network. This should not conflict with any networks to which your system is connected. (Internal and external!)
SUBNET=192.168.99.0/24

#Main mail domain
DOMAIN=MYSITE.us

# Hostnames for this server, separated with commas
HOSTNAMES=mail.MYSITE.us

# Choose how secure connections will behave (value: letsencrypt, cert, notls, mail, mail-letsencrypt)
TLS_FLAVOR=cert
# existing certs should be copied to /certs
#cert.pem
TLS_CERT_FILENAME=local.crt
#key.pem
TLS_KEYPAIR_FILENAME=local.key

RELAYNETS=192.168.86.0/24,192.168.99.0/24

REAL_IP_FROM=192.168.86.0/24,192.168.99.0/24

PROXY_AUTH_WHITELIST=192.168.86.0/24,192.168.99.0/24

# is this possibly causing an issue?  I moved the compose logic to my own monolithic compose file and prepended "mail-" to the containers.
# Docker-compose project name, this will prepended to containers names.
COMPOSE_PROJECT_NAME=mailu

# traefik specific
PROXY_PROTOCOL=25,465,993,995,4190
TRAEFIK_VERSION=v2

# if AVX512 is available
LD_PRELOAD=/usr/lib/libhardened_malloc.so

###end mail.env

The 192.168.86.0/24 is my network.
And 192.168.99.0/24 is the docker network.

traefik.yaml (docker compose, trimmed down)

networks:
  proxy:
    name: proxy
    driver: bridge
    ipam:
      config:
        - subnet: 192.168.99.0/24
  webmail:
    driver: bridge
  clamav:
    driver: bridge
  oletools:
    driver: bridge
    internal: true

services:
  ## traefik proxy
  # https://doc.traefik.io/traefik/setup/docker/
  # htpasswd -nb admin "P@ssw0rd" | sed -e 's/\$/\$\$/g'
  # paste that to the "dashboard-auth.basicauth" line below
  # default admin : P@ssw0rd
  traefik:
    image: traefik:v3.4
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
     # Connect to the 'traefik_proxy' overlay network for inter-container communication across nodes
      proxy:
        ipv4_address: 192.168.99.254
      webmail:
    ports:
      - "80:80"    # http web
      - "443:443"  # https web
      - "8080:8080"  # management
      - "25:25"    # mail smtp
      - "465:465"  # mail submissions
      - "587:587"  # mail smtp tls
      - "993:993"  # mail imap
      - "995:995"  # mail pop3
      - "4190:4190" # mail sieve
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - $DOCKERDIR/appdata/traefik/certs:/certs:ro
      - $DOCKERDIR/appdata/traefik/dynamic:/dynamic:ro
      - $DOCKERDIR/appdata/traefik/logs:/logs
      - $DOCKERDIR/appdata/traefik/letsencrypt:/letsencrypt
    environment:
      TZ: $TZ
      PUID: $PUID
      PGID: $PGID
    command:
      # Access Log
      - "--accesslog=true"
      # Optionally change format or output file (requires volume)
      #- "--accesslog.format=json"  # default common (common (Traefik extended CLF), genericCLF (standard CLF compatible with analyzers), or json)
      - "--accesslog.filepath=/logs/access.log"
      # Optionally filter logs
      #- "--accesslog.filters.statuscodes=400-599"

      # API & Dashboard
      - "--api.dashboard=true"
      - "--api.insecure=false"

      # Certificates - Let's Encrypt configuration
      - "--certificatesresolvers.le.acme.email=MYSITE@gmail.com" # replace with your actual email
      - "--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json"
      ##- "--certificatesresolvers.le.acme.httpchallenge.entrypoint=web"
      - "--certificatesresolvers.le.acme.tlschallenge=true"

      # EntryPoints HTTP
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--entrypoints.web.http.redirections.entrypoint.permanent=true"

      # EntryPoints HTTPS
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.websecure.http.tls=true"

      #EntryPoints MAIL
      - "--entrypoints.smtp.address=:smtp" # 25
      - "--entrypoints.submissions.address=:submission" # 465
      - "--entrypoints.imaps.address=:imaps" # 993
      - "--entrypoints.pop3s.address=:pop3s" # 995
      - "--entrypoints.sieve.address=:sieve" # 4190

      # Plugins
      - "--experimental.plugins.traefik-get-real-ip.modulename=github.com/Paxxs/traefik-get-real-ip"
      - "--experimental.plugins.traefik-get-real-ip.version=v1.0.3"

      # Log
      - "--log.filePath=/logs/log-file.log"
      #- "--log.format=json"  # default common
      - "--log.level=INFO"  # default ERROR (TRACE, DEBUG, INFO, WARN, ERROR, FATAL, and PANIC)
      - "--log.maxBackups=3"  # default 0/infinite
      #- "--log.maxSize=100MB"  # default 100MB

      # Providers
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.docker.network=proxy"

      # Attach the static configuration tls.yaml file that contains the tls configuration settings
      #- "--providers.file.filename=/dynamic/tls.yaml"
      - "--providers.file.directory=/dynamic/"

      # Metrics
      - "--metrics.prometheus=true"
      # If using a dedicated metrics entry point, define it:
      - "--entrypoints.metrics.address=:8082"
      # ... other command arguments ...
      - "--metrics.prometheus=true"
      # Optionally change the entry point metrics are exposed on (defaults to 'traefik')
      - "--metrics.prometheus.entrypoint=metrics"
      # Add labels to metrics for routers/services (can increase cardinality)
      - "--metrics.prometheus.addrouterslabels=true"
      - "--metrics.prometheus.addserviceslabels=true"

    # Traefik Dynamic configuration via Docker labels
    labels:
      # Enable self-routing
      - "traefik.enable=true"

      # Dashboard router
      #- "traefik.http.routers.dashboard.rule=Host(`dashboard.docker.localhost`)"
      - "traefik.http.routers.dashboard.rule=Host(`dashboard.MYSITE.us`)"
      - "traefik.http.routers.dashboard.entrypoints=websecure"
      - "traefik.http.routers.dashboard.service=api@internal"
      - "traefik.http.routers.dashboard.tls=true"

      # Basic-auth middleware
      - "traefik.http.middlewares.dashboard-auth.basicauth.users=admin:SANITIZED."
      - "traefik.http.routers.dashboard.middlewares=dashboard-auth@docker"
      - "traefik.http.routers.dashboard.tls=true"

  # GoAccess Site Visitor Logs (reporting) Included to show a web front-end that IS working
  goaccess:
    ### default admin / admin  ??
    image: xavierh/goaccess-for-nginxproxymanager:latest
    container_name: goaccess
    networks:
      proxy:
        ipv4_address: 192.168.99.189
    ports:
      - "7880:7880"
    volumes:
      - $DOCKERDIR/appdata/traefic/logs:/opt/log
      - $DOCKERDIR/appdata/goaccess/GeoLite2/GeoLite2-City.mmdb:/GeoLite2-City.mmdb
#      - /path/to/host/custom:/opt/custom #optional, required if using log_type = CUSTOM
      - "/etc/localtime:/etc/localtime:ro"
    environment:
      TZ: $TZ
      PUID: $PUID
      PGID: $PGID
      SKIP_ARCHIVED_LOGS: "True"
      DEBUG: "False" #optional
      BASIC_AUTH: "False" #optional
      BASIC_AUTH_USERNAME: "$DOZZLE_USERNAME" #optional
      BASIC_AUTH_PASSWORD: "$DOZZLE_PASSWORD" #optional
      EXCLUDE_IPS: "127.0.0.1" #optional - comma delimited
      LOG_TYPE: "TRAEFIKCLF" #optional - https://goaccess.io/man
    security_opt:
      - no-new-privileges:true
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.goaccess.rule=Host(`goaccess.MYSITE.us`)"
      - "traefik.http.routers.goaccess.entrypoints=websecure"
      - "traefik.http.routers.goaccess.tls=true"
      - "traefik.http.routers.goaccess.tls.certresolver=le"

  # mail-front
  # I know that mailu has a bunch of containers, the only one exposed is mail-front, so the rest are trimmed out.
  mail-front:
    # start from  https://setup.mailu.io/2024.06/  to get yaml file started
    image: ${DOCKER_ORG:-ghcr.io/mailu}/${DOCKER_PREFIX:-}nginx:${MAILU_VERSION:-2024.06}
    container_name: mail_front
    restart: always
    env_file: $DOCKERDIR/appdata/mailu/mailu.env
    logging:
      driver: journald
      options:
        tag: mailu-front
    ports:
      #- "11080:80"  # http
      - "11443:443"  # https
      #- "110:110"  #pop3 (old)
      #- "143:143"  # imap (old)
      #- "25:25"  # smtp
      #- "465:465"  # submission
      #- "587:587"  # smtp tls
      #- "993:993"  # imap
      #- "995:995"  # pop3
      #- "4190:4190"  # sieve (manage sieve, mail filtering rules)
    networks:
      proxy:
        ipv4_address: 192.168.99.233
      webmail:
    #networks:
    #  - proxy
    #  - webmail
    volumes:
      - "$DOCKERDIR/appdata/traefik/certs:/certs"
      - "$DOCKERDIR/appdata/mailu/overrides/nginx:/overrides:ro"
    environment:
      TZ: $TZ
      PUID: $PUID
      PGID: $PGID
      ADMIN_ADDRESS: mail-admin
      FRONT_ADDRESS: mail-front
      SMTP_ADDRESS: mail-smtp
      IMAP_ADDRESS: mail-imap
      OLETOOLS_ADDRESS: mail-oletools
      REDIS_ADDRESS: mail-redis
      ANTIVIRUS_ADDRESS: mail-antivirus
      ANTISPAM_ADDRESS: mail-antispam
      WEBMAIL_ADDRESS: mail-webmail
    depends_on:
      - mail-resolver
    dns:
      - 192.168.99.230
    labels:
      - "traefik.enable=true"
      #- "traefik.docker.network=proxy"

      # the second part is important to ensure Mailu can get certificates from letsencrypt for all hostnames
      - "traefik.http.routers.mailweb.rule=Host(`mail.MYSITE.us`) || PathPrefix(`/.well-known/acme-challenge/`)"
      - "traefik.http.routers.mailweb.entrypoints=web"
      - "traefik.http.routers.mailweb.service=mailweb"
      - "traefik.http.services.mailweb.loadbalancer.server.port=11080"

      ## I tried isolating just the mail host to get the page to load from outside, but no luck
      #- "traefik.http.routers.mailsecure.rule=Host(`mail.MYSITE.us`)"
      #- "traefik.http.routers.mailsecure.entrypoints=websecure"
      #- "traefik.http.routers.mailsecure.tls=true"
      #- "traefik.http.routers.mailsecure.tls.certresolver=le"
      #- "traefik.http.routers.mailsecure.service=mailsecure"
      #- "traefik.http.services.mailsecure.loadbalancer.server.port=11443"

      # other FQDNS can be added here:
      - "traefik.tcp.routers.mailsecure.rule=HostSNI(`mail.MYSITE.us`) || HostSNI(`autoconfig.MYSITE.us`) || HostSNI(`mta-sts.MYSITE.us`)"
      - "traefik.tcp.routers.mailsecure.entrypoints=websecure"
      - "traefik.tcp.routers.mailsecure.tls=true"
      - "traefik.tcp.routers.mailsecure.tls.certresolver=le"
      #- "traefik.tcp.routers.mailsecure.tls.passthrough=true"
      - "traefik.tcp.routers.mailsecure.service=mailsecure"
      - "traefik.tcp.services.mailsecure.loadbalancer.server.port=11443"
      - "traefik.tcp.services.mailsecure.loadbalancer.proxyProtocol.version=2"

      # smtp 25
      - "traefik.tcp.routers.smtp.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.smtp.entrypoints=smtp"
      - "traefik.tcp.routers.smtp.service=smtp"
      - "traefik.tcp.services.smtp.loadbalancer.servers.port=25"
      - "traefik.tcp.services.smtp.loadbalancer.proxyProtocol.version=2"

      # submissions 465
      - "traefik.tcp.routers.submissions.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.submissions.entrypoints=submissions"
      - "traefik.tcp.routers.submissions.service=submissions"
      - "traefik.tcp.services.submissions.loadbalancer.server.port=465"
      - "traefik.tcp.services.submissions.loadbalancer.proxyProtocol.version=2"

      # imap 993
      - "traefik.tcp.routers.imaps.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.imaps.entrypoints=imaps"
      - "traefik.tcp.routers.imaps.service=imaps"
      - "traefik.tcp.services.imaps.loadbalancer.server.port=993"
      - "traefik.tcp.services.imaps.loadbalancer.proxyProtocol.version=2"

      # pop3 995
      - "traefik.tcp.routers.pop3s.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.pop3s.entrypoints=pop3s"
      - "traefik.tcp.routers.pop3s.service=pop3s"
      - "traefik.tcp.services.pop3s.loadbalancer.server.port=995"
      - "traefik.tcp.services.pop3s.loadbalancer.proxyProtocol.version=2"

      # sieve 4190
      - "traefik.tcp.routers.sieve.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.sieve.entrypoints=sieve"
      - "traefik.tcp.routers.sieve.service=sieve"
      - "traefik.tcp.services.sieve.loadbalancer.server.port=4190"
      #- "traefik.tcp.services.sieve.loadbalancer.servers.address=:4190"
      - "traefik.tcp.services.sieve.loadbalancer.proxyProtocol.version=2"
    healthcheck:
      test: ['NONE']

end traefik.yaml

Port 25 is forwarding to mail_front, and it doesn't show up when mail_front is stopped

username@comp:/data5/media/Docker$ docker ps | grep 25/
bb13adf23017 Package nginx · GitHub "/bin/sh -c /start.py" 2 hours ago Up 4 seconds 80/tcp, 110/tcp, 0.0.0.0:25->25/tcp, [::]:25->25/tcp, 0.0.0.0:465->465/tcp, [::]:465->465/tcp, 0.0.0.0:587->587/tcp, [::]:587->587/tcp, 0.0.0.0:993->993/tcp, [::]:993->993/tcp, 0.0.0.0:995->995/tcp, [::]:995->995/tcp, 0.0.0.0:4190->4190/tcp, [::]:4190->4190/tcp, 143/tcp, 0.0.0.0:11443->443/tcp, [::]:11443->443/tcp mail_front

But if I try to talk to port 25 it times out, off to the void.

I can log into webmail visiting https:/ /MYHOST:11443/sso/login. It works fine from the inside.

When I try to visit https:/ /mail.MYSITE.us/webmail I get a "404 page not found", because Traefik it getting the request but not determining where to hand it off to.

The traefik access log shows that it isn't routing.

ANIPADDRESS - - [DATETIME] "GET / HTTP/2.0" 200 760904 "-" "-" 6161 "OTHERSERVICE@docker" "http://192.168.99.216:1234" 0ms
ANIPADDRESS - - [DATETIME] "GET /sso/login HTTP/2.0" 404 19 "-" "-" 6162 "-" "-" 0ms
ANIPADDRESS - - [DATETIME] "GET /webmail HTTP/2.0" 404 19 "-" "-" 2819 "-" "-" 0ms

The "-" (3rd from the last field) should be the service it routed to, like "mail_front@docker" (or maybe mailsercure@docker) to show that it passed the request there.

If I look at the traefik dashboard HTTP Routers I see 2 entries for mail-front.
Host(mail-front-docker) metrics web mailsecure@docker mail-front-docker
Host(mail-front-docker) websecure websecure-mailsecure@docker mail-front-docker

I just figured out that the metrics and web entry is coming from the basic traefik configuration section. Though I’m not sure why those are linked to mailsecure (possibly because HTTP redirects to HTTPS and that’s the service listening on that port).

If I try to hit http:/ /MYHOST:11080/sso/login it does not load. (I’m not concerned about the HTTP port though)

I also noticed that the dashboard lists the Name as websecure-mailsecure@docker, so I tried changing the router and service name to match.

  ## other FQDNS can be added here:
  - "traefik.tcp.routers.websecure-mailsecure.rule=HostSNI(`mail.MYSITE.us`) || HostSNI(`autoconfig.MYSITE.us`) || HostSNI(`mta-sts.MYSITE.us`)"
  - "traefik.tcp.routers.websecure-mailsecure.entrypoints=websecure"
  #- "traefik.tcp.routers.websecure-mailsecure.tls=true"
  #- "traefik.tcp.routers.websecure-mailsecure.tls.certresolver=le"
  - "traefik.tcp.routers.websecure-mailsecure.tls.passthrough=true"
  - "traefik.tcp.routers.websecure-mailsecure.service=mailsecure"
  - "traefik.tcp.services.websecure-mailsecure.loadbalancer.server.port=11443"
  - "traefik.tcp.services.websecure-mailsecure.loadbalancer.proxyProtocol.version=2"

Again, no luck.

If I look at HTTP Services it shows:
mail-front-docker@docker at http:/ /192.168.99.233:25
And it says used by routers (both of the above, metrics/web and websecure).

I don’t know why that is trying to link to the SMTP port rather than the WEBSECURE 443.

I'm getting a bunch of these in the traefik log. It looks like it's trying to get a certificate for each subdomain rather than doing a *.MYSITE.us certificate.
DATETIME ERR Unable to obtain ACME certificate for domains error="unable to generate a certificate for the domains [trilium.MYSITE.us]: acme: error: 429 :: POST :: ``https://acme-v02.api.letsencrypt.org/acme/new-order`` :: urn:ietf:params:acme:error:rateLimited :: too many failed authorizations (5) for "``trilium.MYSITE.us``" in the last 1h0m0s, retry after

DATETIME UTC: see ``https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account``" ACME CA=``https://acme-v02.api.letsencrypt.org/directory`` acmeCA=``https://acme-v02.api.letsencrypt.org/directory`` domains=["``trilium.MYSITE.us``"] providerName=le.acme routerName=trilium@docker rule=Host(trilium.MYSITE.us)

The acme.json has certs for 2 subdomains, neither of those is mail.MYSITE.us.
I only noticed the first one the other day, it may be slowly adding sites.

So I know that something's not right, and that there some mental disconnect between how it works and how I want it to work, but I'm not quite sure where the outage lies.

Edits: Removed some things from the traefik.yaml that I had added while messing around, they made things mad.

4

The post looks a bit unstructured, what are you trying to achieve?

TLS
Which service should issue and use the TLS certs?

In Traefik you use websecure, you have TLS enabled, do you load a custom TLS cert via providers.file?

You create a certResolver, but it seems to be neither assigned to an entrypoint nor to a router, so it will not be used.

You use HostSNI() with a domain name, so a TLS cert needs to exist for Traefik to work, otherwise you need HostSNI(`*`) to match any (and for non TLS ports).

If you do want to use certResolver, you can't really use tls.passthrough=true, because then your target service would need the Traefik generated TLS cert.

Naming

Note that you need a router with name and a service with name, the target service needs to be assigned to the router:

  - "traefik.tcp.routers.websecure-mailsecure.service=mailsecure"
                                                      ^
  - "traefik.tcp.services.websecure-mailsecure.loadbalancer.server.port=11443"
                          ^

Those need to match.

5

I got really close, but never got it quite working.

What I want is for the wildcard cert that I have to take care of everything, and for traefik to control renewing that. I learned a lot along the fight, like the tls challenge doesn’t work with wildcard certs, you need to use dns challenge.

I struggled for a bit with how to specify arrays in the docker yaml file, but once I saw an example that made perfect sense.

I struggled a bit with what could go into a dynamic file vs what should be in the static file (I might have skipped a bit of the early documentation that would have helped there).

But to further explain what I was trying to do, I wanted to set up the mail server Mailu inside my network so that Vaultwarden could send it’s emails there. None of that needs to truly be visible outside. But I figured if I’m setting up a mail box then it would be cool if I could use it, so I wanted it to get it working as webmail from outside.

I could get the inside webmail working but the relayed mail wasn’t showing up when using the self signed cert and bypassing the browser warning. The 80 port was looping and never logging in. I was getting some IP from the Netherlands trying to hit the 25 port with a dictionary attack every 2 minutes, trying to be unnoticed. And could only get as close as a cert failure from the outside.

I could get to where traefik tried to route an incoming request to the “http://InternalIP:CorrectPort” of the mail docker box, but the page would be an error. I forget what it was, but almost like the mail box was not seeing the request as coming from local as an allowed IP. Which I do have set.

As traefik is working for everything except mail I think it’s more of an issue with the mail server than with traefik. That said, it was the only one where I had a loadbalancer port setup to get things through.

I gave up on the mail box and closed the ports in the router. That’s just a headache I don’t need. I didn’t want to tie myself to an external service, and I may try to find a different email container to try out, but for now I need so few emails that if the external service breaks things on me I might not notice for a while.

Thanks for the response. I’m not sure if I can close the post without marking a solution, but feel free to close it.

6

Setting up an email server is always a pain. Especially when you try it with a reverse proxy in front. My solution is to run it in a separate VM with own IP.

Maybe running a different mail-server is easier, found some documentation. Which might also help with mailu.

Having continuous login and hacking attempts from the Internet is normal, from any country in the world. IPs of providers are public, they are continuously probed. And also LetsEncrypt has a public ledger, so bots will hit your server as soon as a new cert is issued. Wildcard may be better then individual sub-domain certs.