Traefik 3 - Only Traefik Dashboard works

I am getting Traefik3 to work only partly.
I have admit that I am complete newbie on Traefik, and just tried to follow these instructions with some changes:

Main difference that I used Portainer - Stacks instead of docker-compose.yaml.
This Traefik is running on Proxmox - Ubuntu 22.04 Server VM, where is just Docker with Portainer and Traefik containers.

https://traefik-dashboard.mydomain/ is working fine, connection is secure and certificate looks ok to me.

But all other services or such (Proxmox, Homeassistant) running on different VMs / Servers does not work.

  • https://proxmox.mydomain/ => I am getting "404 page not found" error, but connection is secure.
  • https://homeassistant.mydomain/ => I am getting "400: Bad Request". Also got "Internal Server Error" after small modification on config.yml. Homeassistant is HAOS VM install. Also secure connection.

Debug logs did not show any errors or warnings. I think port handling does not work correctly. Proxmox is using 8006 and Homeassistant 8123.

I have defined those DNS names to PiHole and using Traefik IP instead of actual VM IP.

Is some additional configuration needed when service is not running on Same VM (same IP) as Traefik?

I don't need access service outside of my local network at the moment, so I have not defined any DNS entry on Cloudflare.

Share your full Traefik static and dynamic config, and docker-compose.yml or equivalent if used.

I have been trying to get work only js-zen-prox Proxmox server, homeassistant and truenas.
mydomain.net is fake here, but I have reserved own xxxx.net domain from CloudFlare that I am using also below configurations.


#### Portainer Stacks: ####
version: '3'

services:
  traefik:
    image: traefik:latest
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - proxy
    ports:
      - 80:80
      - 443:443

    environment:
      - CF_DNS_API_TOKEN=xxxxx
      - CF_API_EMAIL=xxxx@gmail.com
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /home/js/docker_volumes/traefik/data/traefik.yml:/traefik.yml:ro
      - /home/js/docker_volumes/traefik/data/acme.json:/acme.json
      - /home/js/docker_volumes/traefik/data/config.yml:/config.yml:ro
      - /home/js/docker_volumes/trafik/data/logs:/var/log/traefik
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`traefik-dashboard.mydomain.net`)"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=xxxx"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`traefik-dashboard.mydomain.net`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=cloudflare"
      - "traefik.http.routers.traefik-secure.tls.domains[0].main=mydomain.net"
      - "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.mydomain.net"
      - "traefik.http.routers.traefik-secure.service=api@internal"
      #- "traefik.http.middlewares.my-service-ipallowlist.ipallowlist.sourcerange=192.168.50.0/24"
      #- "traefik.http.routers.my-service.middlewares=my-service-ipallowlist@docker"

networks:
  proxy:
    external: true

#### traefik.yml: ####
api:
  dashboard: true
  debug: true

entryPoints:
  http:
    address: ":80"
    http:
      redirections:
        entrypoint:
          to: https
          scheme: https

  https:
    address: ":443"

serversTransport:
  insecureSkipVerify: false

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
  file:
    filename: /config.yml
    watch: true

log:
  level: "DEBUG"
  filePath: "/var/log/traefik/traefik.log"
accessLog:
  filePath: "/var/log/traefik/access.log"

certificatesResolvers:
  cloudflare:
    acme:
      email: xxxx@gmail.com
      storage: acme.json
      caServer: https://acme-v02.api.letsencrypt.org/directory # production
      dnsChallenge:
        provider: cloudflare
        #disablePropagationCheck: true # uncomment this if you have issues pulling certificates through cloudflare, By>
        #delayBeforeCheck: 60s # uncomment along with disablePropagationCheck if needed to ensure the TXT record is re>
        resolvers:
          - "1.1.1.1:53"
          - "1.0.0.1:53"

#### config.yml ####
http:
 #region routers
  routers:
    js_zen_proxmox:
      entryPoints:
        - "https"
      rule: "Host(`js-zen-prox.mydomain.net`)"
      middlewares:
        - default-headers
        - https-redirectscheme
      tls: {}
      service: js_zen_proxmox

    homeassistant:
      # For Homeassistant config, check: https://www.home-assistant.io/integrations/http/#reverse-proxies
      # This relies on Homeassistant using http. No certs are needed in the Homeassistant config.
      entryPoints:
        - "https"
      rule: "Host(`homeassistant.mydomain.net`)"
      middlewares:
        - default-headers
        - https-redirectscheme
      tls: {}
      service: homeassistant

    truenas:
      entryPoints:
        - "https"
      rule: "Host(`truenas.mydomain.net`)"
      middlewares:
        - default-headers
        - https-redirectscheme
      tls: {}
      service: truenas
#endregion

#region services
  services:
    js_zen_proxmox:
      loadBalancer:
        servers:
          - url: "https://192.168.50.130:8006"
        passHostHeader: true

    homeassistant:
      loadBalancer:
        servers:
          - url: "https://192.168.50.60:8123"
        passHostHeader: true

    truenas:
      loadBalancer:
        servers:
          - url: "https://192.168.50.85"
        passHostHeader: true

#endregion

  middlewares:
    https-redirectscheme:
      redirectScheme:
        scheme: https
        permanent: true

    default-headers:
      headers:
        frameDeny: true
        browserXssFilter: true
        contentTypeNosniff: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 15552000
        customFrameOptionsValue: SAMEORIGIN
        customRequestHeaders:
          X-Forwarded-Proto: https

    default-whitelist:
      ipAllowList:
        sourceRange:
        - "192.168.50.0/24"

[quote="bluepuma77, post:2, topic:22586, full:true"]
Share your full Traefik static and dynamic config, and `docker-compose.yml` or equivalent if used.
[/quote]

You mount traefik.yml to /. Use target /etc/traefik/traefik.yml (doc) or add

command: --configFile=/traefik.yml

in docker-compose.yml (doc).

Also

storage: acme.json

should use an absolute path (/acme.json) for the mounted file.

I could not add command: or storage: to stacks -> it did not start then.
However, I got working all services that won't require port on address field:

services:
proxmoxzen:
loadBalancer:
servers:
- url: "http://192.168.50.130:8006"
passHostHeader: true

truenas:
  loadBalancer:
    servers:
      - url: "http://192.168.50.85"
    passHostHeader: true

homeassistant:
  loadBalancer:
    servers:
      - url: "http://192.168.50.60:8123"
    passHostHeader: true

For example: Truenas is working fine via Traefik, but ProxmoxZen and Homeassistant does not.
I am getting different errors on those;
Homeassistant: 400: Bad Request
ProxmoxZen: ERR_TOO_MANY_REDIRECTS

I can concur that with Traefik 3, Home Assistant does not work, I get http 404
Every other service is working in my case except Home Assistant.
If I revert back to an older version (the latest before 3) it works fine.

Check and compare to simple Traefik example.

If your configuration is not working, you should share Traefik static and dynamic config, config of the target service, and docker-compose.yml if used.

Found the issue:

error while adding rule Host(`homeassistant.REDACTED.com`, `haos.REDACTED.com`): error while adding rule Host: unexpected number of parameters; got 2, expected one of [1]

And it's not documented in:

However this fixes it:

core:
  defaultRuleSyntax: v2

Or: Host rule does not support multiple hostnames anymore

I have got working all services that I wanted, except homeassistant.
I tried to add this core: rule, but not sure where to configure it.

Can you share full homeassistant config?

And also if I try to enable homeassistant to config.yml, also all other services stops to work (homeassistant is defined exactly same way as all other services)

Edit: I did minor change and now it is not crashing other services. On Traefik Dashboard Homeassistant looks fine. But when trying to access HA, I will get 'Internal Server Error'

Share your full Traefik static and dynamic config, and docker-compose.yml or equivalent if used.

Was it working with v2?
If you never used v2, you don't need that core rule.
In my case it was working for v2 but broke in v3 because of host rule format for multiple hostnames. (See the link above)

I just started to use Traefik on version 3. No past experience.

then get rid of that v2 compatibility part.
As requested by @bluepuma77 share your config.

Do you have traefik IP added to trusted_proxies?

http:
  use_x_forwarded_for: true
  trusted_proxies:
    - 192.168.1.130/32 # Host IP where Traefik container is running
    - 172.23.0.2/32 # Container IP of Traefik