Internal server error when trying to connect via proxy to app

Traefik Traefik v2
1

I am getting Internal Server Error when trying to connect to portainer via my domain + path prefix.

Here is my docker-compose.traefik.yml

version: "3.3"

services:

  traefik:
    image: traefik:v2.6
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - traefik_proxy
      - internal
    ports:
      - 80:80
      - 443:443
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /share/docker/traefik/config/traefik.yml:/traefik.yml:ro
      - /share/docker/traefik/config/.htpasswd:/.htpasswd
      - /share/docker/traefik/config/traefik-dynamic.yml:/traefik-dynamic.yml:ro
      - /share/docker/traefik/acme/acme.json:/acme.json
      - /share/docker/traefik/logs:/var/log/traefik

  whoami:
    image: "traefik/whoami"
    container_name: "whoami"
    restart: unless-stopped
    networks:
      - internal
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`<mydomain>`) && PathPrefix(`/whoami`)"
      - "traefik.http.routers.whoami.entrypoints=websecure"

# ## run
# ## docker network create traefik_proxy
# ## to create the network
networks:
  traefik_proxy:
    external: true
  internal:

and my static config traefik.yml

# ## STATIC CONFIGURTATION
# ##  https://doc.traefik.io/traefik/reference/static-configuration/file/
# ##
# ## do not mix with the dynamic configuration
# ##  https://doc.traefik.io/traefik/reference/dynamic-configuration/file/
# ##  or in docker format
# ##  https://doc.traefik.io/traefik/reference/dynamic-configuration/docker/

global:
  checkNewVersion: true
  sendAnonymousUsage: false  # true by default

# ## (Optional) Log information
# ## https://doc.traefik.io/traefik/observability/logs/
log:
  level: DEBUG  # DEBUG, INFO, WARNING, ERROR, CRITICAL
  format: common  # common, json, logfmt
  #filePath: /var/log/traefik/traefik.log

# ## (Optional) Accesslog
# ## https://doc.traefik.io/traefik/observability/access-logs/
accesslog:
  format: common  # common, json, logfmt
  #filePath: /var/log/traefik/access.log

# ## (Optional) Enable API and Dashboard
# ## https://doc.traefik.io/traefik/operations/dashboard/
# ## https://doc.traefik.io/traefik/operations/api/
# ## API endpoints: https://doc.traefik.io/traefik/operations/api/#endpoints
api:
  dashboard: true  # true by default
  insecure: false
  #debug: true

# ## https://doc.traefik.io/traefik/operations/ping/
ping:
  entryPoint: web

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
  # ## https://doc.traefik.io/traefik/providers/file/
  # ## ensure the specified filename / directory is mounted in volumes in the docker-compose file
  file:
    filename: "/traefik-dynamic.yml"
    #directory: /dynamic
    watch: true

# ## https://doc.traefik.io/traefik/routing/entrypoints/
entryPoints:
  web:
    address: ":80"
    # ## https://doc.traefik.io/traefik/routing/entrypoints/#http-options
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
          permanent: true
  websecure:
    address: ":443"
    http:
      tls:
        certResolver: letsencrypt

# ## https://doc.traefik.io/traefik/routing/overview/
# ## https://doc.traefik.io/traefik/routing/overview/#insecureskipverify
#serversTransport:
#  insecureSkipVerify: true

# ## https://doc.traefik.io/traefik/https/acme/
# ## https://doc.traefik.io/traefik/user-guides/docker-compose/acme-http/
# ## Note, Let's Encrypt rate limits
# ##  https://letsencrypt.org/docs/rate-limits/
certificatesResolvers:
  letsencrypt:
    acme:
      email: <mymail>
      # ## uncomment the following when testing
      # ## see also https://doc.traefik.io/traefik/user-guides/docker-compose/acme-http/
      #caServer: https://acme-staging-v02.api.letsencrypt.org/directory
      storage: acme.json
      httpChallenge:
        entryPoint: web

My dynamic config traefik-dynamic.yml

enable: true

http:
  routers:
    dashboard:
      # ## IP not working with let's encrypt!
      rule: Host(`<mydomain>`)
      service: api@internal
      entryPoints:
        - websecure
      middlewares:
        - auth
    ping:
      rule: Host(`<mydomain>`) && PathPrefix(`/ping`)
      service: ping@internal
      entryPoints:
        - websecure

  middlewares:
    auth:
      basicAuth:
        usersFile: "/.htpasswd"

and finally my docker-compose.portainer.yml

version: '3'

services:
  portainer:
    container_name: portainer
    image: portainer/portainer-ce:2.11.1
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - traefik_proxy
    ports:
      - "8000:8000"
      #- "9000:9000"
      - "9443:9443"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock
      - /share/docker/portainer/data:/data
    labels:
      - "traefik.enable=true"
      #- "traefik.docker.network=traefik_proxy"

      # ## https://github.com/portainer/portainer/issues/6337
      #- "traefik.http.routers.portainer-secure.tls=true"
      #- "traefik.http.services.portainer.loadbalancer.server.scheme=https"

      # ## https://docs.portainer.io/v/ce-2.11/advanced/reverse-proxy/traefik

      # ## Frontend
      - "traefik.http.routers.portainer-frontend.entrypoints=websecure"
      - "traefik.http.routers.portainer-frontend.rule=Host(`<mydomain>`) && PathPrefix(`/portainer`)"
      - "traefik.http.routers.portainer-frontend.tls=true"
      - "traefik.http.routers.portainer-frontend.service=frontend"
      - "traefik.http.services.frontend.loadbalancer.server.port=9443"
      - "traefik.http.services.frontend.loadbalancer.server.scheme=https"
      #- "traefik.http.routers.frontend.tls.certresolver=letsencrypt"
      # ## Edge
      #- "traefik.http.routers.portainer-edge.entrypoints=websecure"
      #- "traefik.http.routers.portainer-edge.rule=Host(`<mydomain>`) && PathPrefix(`/portainer/edge`)"
      #- "traefik.http.routers.portainer-edge.service=edge"
      #- "traefik.http.services.edge.loadbalancer.server.port=8000"

networks:
  traefik_proxy:
    external: true

I can access portainer locally but when I try to access via <mydomain>/portainer I get

Internal Server Error

and in the traefik log I see

level=debug msg="'500 Internal Server Error' caused by: x509: certificate is valid for 0.0.0.0, not 172.29.8.3"

which seems already pretty obvious that the issue is a certificate.

Obviously I am pretty new to portainer and probably lack some basics here.
Anyways, to my understanding I configured traefik such that every incoming connection using the entryPoints web or websecure to <mydomain> (and also <mydomain>/*) is going via the https using LetsEncrypt certificates for the connection.

Further, the transport from traefik to the app (here portainer) is then handled via serverTransport (https://doc.traefik.io/traefik/routing/services/) which to my understanding is also where the problem is.

Unfortunately I am unable to figure out how to configure portainer such that this issue goes away (without getting a 404...).

As I see it, I have the following options:

  • have traefik connect via http to the app (here portainer) (this should be fine because it is in my local network only)
  • have traefik use a self-signed certificate (e.g. the one generated by portainer)
  • tell traefik to skip verification via insecureSkipVerify

So my questions are:

  • Where do I have to / can place this configuration?
  • How would the configuration for use of the self-signed portainer certificate look like?

Finally, I would prefer to have all portainer - traefik related config in one place, either in a dynamic config, e.g. traefik-portainer.yml file or in my docker-compose.portainer.yml.
I guess there is at least some basic configuration required in the docker compose file, i.e. - "traefik.enable=true" everything else could go in a dedicated dynamic configuration, right?

I appreciate any comments / hints / help.

2

In the meantime I also tried to set

      - "traefik.http.services.frontend.loadbalancer.serversTransport=my"
      - "traefik.http.services.frontend.loadbalancer.serversTransport.my.insecureSkipVerify=true"

in docker-compose.portainer.yml but this leads to 404 page not found with

level=error msg="invalid node serversTransport: string"

in the traefik.log

3

Also when trying to have traefik connect to portainer via http (port 9000), i.e. the labels in my docker-compose.portainer.yml are

    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.portainer-frontend.entrypoints=websecure"
      - "traefik.http.routers.portainer-frontend.rule=Host(`<mydomain>`) && PathPrefix(`/portainer`)"
      - "traefik.http.routers.portainer-frontend.service=frontend"
      - "traefik.http.services.frontend.loadbalancer.server.port=9000"

but I get 404 page not found.
In the traefik logs everything looks fine, the portainer server is up and running (with a different IP compared to the whoami server which is on the internal network).

Also adding

      - "traefik.docker.network=traefik_proxy"

changes nothing...