Traefik can't proxy Heimdall - 404 error

Traefik Traefik v2 (latest)
1

I'm having an odd issue with Traefik v2.5.3.

Running under Portainer v2 and have about 12 containers, 7 of them are exposed via Traefik and they're all working fine. I'm trying to add Heimdall which is a much simpler app than everything else but it isn't working.

I'm getting a 404 externally but if I check curl locally on the docker container it's working fine.

Service, Router all look good, the forwarding IP/port is correct.

Traefik config (The certs are from a local Hashicorp Vault CA server, self hosted)

global:
  checkNewVersion: true
  sendAnonymousUsage: false  # true by default
serversTransport:
  rootCAs:
    - ca_cert.pem
    - ca_cert.crt
log:
  level: DEBUG  
api:
  dashboard: true
  insecure: true
ping: {}
entryPoints:
  web:
    address: :80
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https

  websecure:
    address: :443
    forwardedHeaders:
       trustedIPs=192.168.1.0/24

providers:
  docker:
    exposedByDefault: true  # Default is true
    watch: true

Labels, being applied:

traefik.enable: true 
traefik.frontend.certFile: /home/portainer/certs/heimdall_cert.pem 
traefik.frontend.keyFile: /home/portainer/certs/heimdall_key.pem
traefik.http.routers.heimdall.entrypoints: web,websecure
traefik.http.routers.heimdall.rule: Host(`heimdall.$DOMAIN`)
traefik.http.routers.heimdall.tls: true
traefik.http.services.heimdall-service.loadbalancer.server.port: 80

Command line:

docker$ curl -k https://heimdall
404 page not found
docker$ docker exec -it heimdall curl http://0.0.0.0 | head
<!doctype html>
<html lang="en">....
2

I'm running in to this same exact issue for about a week now. No matter what I try I have the same problem. In addition, starting Heimdall kills the Traefik dashboard as well with a 404, but I cannot find errors in any logs anywhere. Everything else keeps running fine.

Did you ever find a solution?

3

It sort of started working I have no idea what I changed to make it work but it did. The only thing I have different for heimdall vs the other containers is that I have a specific SSL cert for heimdall.mydomain.com vs. all the others use a wildcard cert.

I'll include my config in case it helps:
Here are my heimdall container labels:

  "traefik.enable": "true",
  "traefik.http.routers.heimdall.tls": "true",
  "traefik.network": "traefik_default"

In my traefik.yml (mapped) I have:

providers:
  file:
    directory: "/etc/traefik/dyn/"
  docker:
    defaultRule: "Host(`{{ normalize .Name }}.mydomain.com`)"
    endpoint: unix:///var/run/docker.sock
    exposedByDefault: true
    watch: true

In dyn.yml (mapped), I have:

tls:
  options:
    default:
      minVersion: VersionTLS12

  certificates:
    - certFile: /certs/traefik_cert.pem
      keyFile: /certs/traefik_key.pem

    - certFile: /certs/heimdall_cert.pem
      keyFile: /certs/heimdall_key.pem

    - certFile: /certs/wildcard_cert.pem
      keyFile: /certs/wildcard_key.pem