Portainer not working under Traefik

Hello, I've followed a quite recent Traefik's setup guide (2024) and I can reach correctly its dashboard from outside but when I've included Portainer under Traefik's network and tried to reach it from external addresses, I got this error from the page https://portainer.xxx.home/ :

Server Internal error

and if I try to reach https://portainer.xxx.home:9000/

I get this:
This site can’t be reached
portainer.xxx.home refused to connect.

ERR_CONNECTION_REFUSED

In Traefik's dashboard there is no trace of error. I'm totally new to this, any clue? Thanks in advance!

This is my portainer.yml:

services:
  # Portainer - WebUI for Containers
  portainer:
    container_name: portainer
    image: portainer/portainer-ce:latest # Use portainer-ce for the rest of us
    security_opt:
      - no-new-privileges:true
    restart: unless-stopped
    # profiles: ["core", "all"]
    networks:
        - socket_proxy
        - t3_proxy
    # command: -H unix:///var/run/docker.sock # # Use Docker Socket Proxy instead for improved security
    command: -H tcp://socket-proxy:2375
    ports:
      - "9000:9000"
    volumes:
      - $DOCKERDIR/appdata/portainer/data:/data 
    environment:
      - TZ=$TZ
    labels:
      - "traefik.enable=true"
      # HTTP Route
      - "traefik.http.routers.portainer-rtr.entrypoints=websecure"
      - "traefik.http.routers.portainer-rtr.rule=Host(`portainer.$DOMAINNAME_1`)"
      # Middlewares
      - "traefik.http.routers.portainer-rtr.middlewares=chain-no-auth@file" # Portainer does not play well with Basic HTTP Authentication. Plus, it has built-in authentication. 
      # HTTP Services
      - "traefik.http.routers.portainer-rtr.service=portainer-svc"
      - "traefik.http.services.portainer-svc.loadbalancer.server.port=9000"
      - "traefik.http.services.portainer-svc.loadbalancer.server.scheme=https"

and this is my traefik.yml:

services:
  # Traefik 3 - Reverse Proxy
  traefik:
    container_name: traefik
    image: traefik:latest
    security_opt:
      - no-new-privileges:true
    restart: unless-stopped
    # profiles: ["core", "all"]
    networks:
      t3_proxy:
        ipv4_address: 192.xxx.xxx.254 # You can specify a static IP
      socket_proxy:
    command: # CLI arguments
      - --global.checkNewVersion=false
      - --global.sendAnonymousUsage=true
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
      - --entrypoints.traefik.address=:8080
      - --entrypoints.websecure.http.tls=true
      - --entrypoints.web.http.redirections.entrypoint.to=websecure
      - --entrypoints.web.http.redirections.entrypoint.scheme=https
      - --entrypoints.web.http.redirections.entrypoint.permanent=true
      - --api=true
      - --api.dashboard=true
      - --api.insecure=true
      #- --serversTransport.insecureSkipVerify=true
      # Allow these IPs to set the X-Forwarded-* headers - Cloudflare IPs: https://www.cloudflare.com/ips/
      #- --entrypoints.websecure.forwardedHeaders.trustedIPs=$CLOUDFLARE_IPS,$LOCAL_IPS
      - --log=true
      - --log.filePath=/logs/traefik.log
      - --log.level=ERROR # (Default: error) DEBUG, INFO, WARN, ERROR, FATAL, PANIC
      - --accessLog=true
      - --accessLog.filePath=/logs/access.log
      - --accessLog.bufferingSize=100 # Configuring a buffer of 100 lines
      - --accessLog.filters.statusCodes=204-299,400-499,500-599
      - --providers.docker=true
      # - --providers.docker.endpoint=unix:///var/run/docker.sock # Disable for Socket Proxy. Enable otherwise.
      - --providers.docker.endpoint=tcp://socket-proxy:2375 # Enable for Socket Proxy. Disable otherwise.
      - --providers.docker.exposedByDefault=false
      - --providers.docker.network=t3_proxy 
      # - --providers.docker.swarmMode=false # Traefik v2 Swarm
      # - --providers.swarm.endpoint=tcp://127.0.0.1:2377 # Traefik v3 Swarm
      - --entrypoints.websecure.http.tls.options=tls-opts@file
      # Add dns-cloudflare as default certresolver for all services. Also enables TLS and no need to specify on individual services
      - --entrypoints.websecure.http.tls.certresolver=dns-cloudflare
      - --entrypoints.websecure.http.tls.domains[0].main=$DOMAINNAME_1
      - --entrypoints.websecure.http.tls.domains[0].sans=*.$DOMAINNAME_1
      # - --entrypoints.websecure.http.tls.domains[1].main=$DOMAINNAME_2 # Pulls main cert for second domain
      # - --entrypoints.websecure.http.tls.domains[1].sans=*.$DOMAINNAME_2 # Pulls wildcard cert for second domain
      - --providers.file.directory=/rules # Load dynamic configuration from one or more .toml or .yml files in a directory
      - --providers.file.watch=true # Only works on top level files in the rules folder
      # - --certificatesResolvers.dns-cloudflare.acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory # LetsEncrypt Staging Server - uncomment when testing
      - --certificatesResolvers.dns-cloudflare.acme.storage=/acme.json
      - --certificatesResolvers.dns-cloudflare.acme.dnsChallenge.provider=cloudflare
      - --certificatesResolvers.dns-cloudflare.acme.dnsChallenge.resolvers=1.1.1.1:53,1.0.0.1:53
      - --certificatesResolvers.dns-cloudflare.acme.dnsChallenge.delayBeforeCheck=90 # To delay DNS check and reduce LE hitrate
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
    # Enable these following 4 lines to have Traefik reachable from the inside in plain http mode
      - target: 8080 # need to enable --api.insecure=true
        published: 8080
        protocol: tcp
        mode: host
    volumes:
      - $DOCKERDIR/appdata/traefik3/rules/$HOSTNAME:/rules # Dynamic File Provider directory
      # - /var/run/docker.sock:/var/run/docker.sock:ro # Enable if not using Socket Proxy
      - $DOCKERDIR/appdata/traefik3/acme/acme.json:/acme.json # Certs File 
      - $DOCKERDIR/logs/$HOSTNAME/traefik:/logs # Traefik logs
    environment:
      - TZ=$TZ
      - CF_DNS_API_TOKEN_FILE=/run/secrets/cf_dns_api_token    
      - HTPASSWD_FILE=/run/secrets/basic_auth_credentials # HTTP Basic Auth Credentials
      - DOMAINNAME_1 # Passing the domain name to traefik container to be able to use the variable in rules. 
    secrets:
      - cf_dns_api_token
      - basic_auth_credentials
    labels:
      - "traefik.enable=true"
      # HTTP Routers
      - "traefik.http.routers.traefik-rtr.entrypoints=websecure"
      - "traefik.http.routers.traefik-rtr.rule=Host(`traefik.$DOMAINNAME_1`)"
      # Services - API
      - "traefik.http.routers.traefik-rtr.service=api@internal"
      # Middlewares
      - "traefik.http.routers.traefik-rtr.middlewares=chain-basic-auth@file" # For Basic HTTP Authentication
      - "traefik.http.services.portainer-svc.loadbalancer.server.port=9000"
      - "traefik.http.services.portainer-svc.loadbalancer.server.scheme=https"

Portainer port 9000 should be http. Port 9443 should be https, but probably needs insecureSkipVerify.

First step is to enable Traefik debug log (doc) and also Traefik access log in JSON format (doc). Check if those requests arrive at Traefik.

- --serversTransport.insecureSkipVerify=true makes things working properly. But what about its side effect? Is it dangerous?

I will enable the debug log. But one more question: all this is happening because Portainer normally is using its own certificates for https?
Indeed is there any way to pass my certificates to Portainer, provided that those certificates are generated on-the.fly and changed all the times?

Portainer by itself can only create a custom cert, which is not trusted by Traefik.

Options:

  1. Use http
  2. Use https with insecureskipverify
  3. Use a manully created custom cert which you import in Traefik and Portainer

It depends on your network security and overall security requirements and risk profile.