Configuring Traefik in Portainer/Docker with HTTPS for external- and internal communication with Kestrel

Hi! We are new to Traefik and are currently facing a configuration challenge involving secure external and internal communication with Kestrel. Our setup is as follows:

Traefik is deployed within Portainer, and we possess a wildcard certificate (*.mycompany.com) signed by our company's Root CA (Root CA A). The containers running in Portainer are .NET applications labeled with the following Traefik settings:

        - "traefik.enable=true"
        - "traefik.http.routers.app1.rule=Host(`app1.mycompany.com`)"
        - "traefik.http.routers.app1.entrypoints=websecure"
        - "traefik.http.routers.app1.tls=true"
        - "traefik.http.services.app1.loadbalancer.server.scheme=https"
        - "traefik.http.services.app1.loadbalancer.server.port=443"

traefik.yml

serversTransport:
  insecureSkipVerify: true

api:
  dashboard: true
  insecure: true

entryPoints:
  web:
    address: ":80"
  websecure:
    address: ":443"

providers:
  docker:
    swarmMode: true
    exposedByDefault: false
    network: public
  file:
    filename: /etc/traefik/settings/dynamic.yml

dynamic.yml (The certificate file is the aforementioned wildcard certificate.)

tls:
  certificates:
    - certFile: /certs/traefik.cer
      keyFile: /certs/traefik.key
  stores:
    default:
      defaultCertificate:
        certFile: /certs/traefik.cer
        keyFile: /certs/traefik.key

compose.yml (Stack)

version: "3.7"

services:
  traefik:
    image: traefik:latest
    restart: unless-stopped
    hostname: traefik
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
    configs:
      - source: traefik.yml
        target: /config/traefik.yml
      - source: traefik_dynamic.yml
        target: /etc/traefik/settings/dynamic.yml
    command:
      - "--configFile=/config/traefik.yml"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock
      - /docker-data/certs:/certs

    networks:
      - public
    deploy:
      placement:
        constraints: [node.role == manager]
      labels:
        - "traefik.enable=true"
        - "traefik.http.services.traefik.loadbalancer.server.port=8080"
        - "traefik.http.routers.api.rule=Host(`traefik.mycompany.com`)"
        - "traefik.http.routers.api.service=api@internal"
        - "traefik.http.routers.api.middlewares=auth"
        - "traefik.http.middlewares.auth.basicauth.users=traefik:<key>"
configs:
  traefik.yml:
    external: true
  traefik_dynamic.yml:
    external: true
networks:
  public:
    external: true

The wildcard certificate is provided by our corporate instance, and we cannot control it. However, internally (within our Portainer environment), we have a different Root CA, which we'll refer to as Root CA B. When we aim to communicate securely between containers, we use certificates signed by Root CA B, which is where the issue arises.

Currently, communication works both internally and externally with the insecureSkipVerify: true setting in traefik.yml. However, we want to avoid using this setting to ensure the entire chain is secure. Without insecureSkipVerify: true , external access via a browser using "app1.mycompany.com" results in errors like 'Internal Server Error' or 'Too many redirects'.

Our applications are .NET apps configured with Kestrel, which are configured as follows:

{
  "Kestrel": {
    "Endpoints": {
      "Http": {
        "Url": "http://[::]:80"
      },
      "Https": {
        "Url": "https://[::]:443",
        "Certificate": {
          "Path": "./https/cert_signed_by_root_ca_b.pfx",
          "Password": "MySuperSecurePassword!"
        }
      }
    }
  }
}

How can we resolve this issue such that the whole chain is secured? We feel like we’re almost there. Any help is appreciated. If you need more information, let me know! Thanks in advance.

How can it result in different errors? Did you enable and check Traefik debug log (doc)?

You're right, the error we are getting when setting the insecureSkipVerify to false results in: Internal Server Error.

The logs show the following:

level=info msg="Configuration loaded from file: /config/traefik.yml"
level=error msg="accept tcp [::]:80: use of closed network connection" entryPointName=web
level=error msg="accept tcp [::]:443: use of closed network connection" entryPointName=websecure
level=error msg="Error while starting server: accept tcp [::]:80: use of closed network connection" entryPointName=web
level=error msg="accept tcp [::]:8080: use of closed network connection" entryPointName=traefik
level=error msg="close tcp [::]:443: use of closed network connection" entryPointName=websecure
level=error msg="close tcp [::]:8080: use of closed network connection" entryPointName=traefik

Note that some other services are still running on port 80, but they are not relevant for this case.

Are you sure those error are during startup/running? I thought those usually appear when shutting down the container.

Set log level to debug.

Ehhh yes you are right, good thing it's still morning, should have refreshed Portainer before accessing the 'active' container. Here are the logs (filtered to leave out sensitive data), the last entry is from when I tried to access it from the webbrowser:

time="2023-11-07T08:26:30+01:00" level=info msg="Configuration loaded from file: /config/traefik.yml"
time="2023-11-07T08:26:30+01:00" level=info msg="Traefik version 2.10.5 built on 2023-10-11T13:54:02Z"
time="2023-11-07T08:26:30+01:00" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"mqttsecure\":{\"address\":\":8883\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"traefik\":{\"address\":\":8080\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"web\":{\"address\":\":80\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"websecure\":{\"address\":\":443\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}}},\"providers\":{\"providersThrottleDuration\":\"2s\",\"docker\":{\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"swarmMode\":true,\"network\":\"frontend\",\"swarmModeRefreshSeconds\":\"15s\"},\"file\":{\"watch\":true,\"filename\":\"/etc/traefik/settings/dynamic.yml\"}},\"api\":{\"insecure\":true,\"dashboard\":true},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"}}"
time="2023-11-07T08:26:30+01:00" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
time="2023-11-07T08:26:30+01:00" level=info msg="Starting provider aggregator aggregator.ProviderAggregator"
time="2023-11-07T08:26:30+01:00" level=debug msg="Starting TCP Server" entryPointName=web
time="2023-11-07T08:26:30+01:00" level=debug msg="Starting TCP Server" entryPointName=websecure
time="2023-11-07T08:26:30+01:00" level=debug msg="Starting TCP Server" entryPointName=traefik
time="2023-11-07T08:26:30+01:00" level=debug msg="Starting TCP Server" entryPointName=mqttsecure
time="2023-11-07T08:26:30+01:00" level=info msg="Starting provider *file.Provider"
time="2023-11-07T08:26:30+01:00" level=debug msg="*file.Provider provider configuration: {\"watch\":true,\"filename\":\"/etc/traefik/settings/dynamic.yml\"}"
time="2023-11-07T08:26:30+01:00" level=info msg="Starting provider *traefik.Provider"
time="2023-11-07T08:26:30+01:00" level=debug msg="*traefik.Provider provider configuration: {}"
time="2023-11-07T08:26:30+01:00" level=info msg="Starting provider *acme.ChallengeTLSALPN"
time="2023-11-07T08:26:30+01:00" level=debug msg="*acme.ChallengeTLSALPN provider configuration: {}"
time="2023-11-07T08:26:30+01:00" level=info msg="Starting provider *docker.Provider"
time="2023-11-07T08:26:30+01:00" level=debug msg="*docker.Provider provider configuration: {\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"swarmMode\":true,\"network\":\"frontend\",\"swarmModeRefreshSeconds\":\"15s\"}"
time="2023-11-07T08:26:30+01:00" level=debug msg="Configuration received: {\"http\":{},\"tcp\":{},\"udp\":{},\"tls\":{\"stores\":{\"default\":{}}}}" providerName=file
time="2023-11-07T08:26:30+01:00" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"api\":{\"entryPoints\":[\"traefik\"],\"service\":\"api@internal\",\"rule\":\"PathPrefix(`/api`)\",\"priority\":2147483646},\"dashboard\":{\"entryPoints\":[\"traefik\"],\"middlewares\":[\"dashboard_redirect@internal\",\"dashboard_stripprefix@internal\"],\"service\":\"dashboard@internal\",\"rule\":\"PathPrefix(`/`)\",\"priority\":2147483645}},\"services\":{\"api\":{},\"dashboard\":{},\"noop\":{}},\"middlewares\":{\"dashboard_redirect\":{\"redirectRegex\":{\"regex\":\"^(http:\\\\/\\\\/(\\\\[[\\\\w:.]+\\\\]|[\\\\w\\\\._-]+)(:\\\\d+)?)\\\\/$\",\"replacement\":\"${1}/dashboard/\",\"permanent\":true}},\"dashboard_stripprefix\":{\"stripPrefix\":{\"prefixes\":[\"/dashboard/\",\"/dashboard\"]}}},\"serversTransports\":{\"default\":{\"maxIdleConnsPerHost\":200}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=internal
time="2023-11-07T08:26:30+01:00" level=debug msg="No store is defined to add the certificate MIIIgjCCBmqgAwIBAgITRQADzZmlO57mTGHzYgAAAAPNmTANBg, it will be added to the default store."
time="2023-11-07T08:26:30+01:00" level=debug msg="Adding certificate for domain(s) *.mycompany.com,docker.mycompany.com,server123.mycompany.com"
time="2023-11-07T08:26:30+01:00" level=debug msg="Added outgoing tracing middleware api@internal" middlewareType=TracingForwarder entryPointName=traefik routerName=api@internal middlewareName=tracing
time="2023-11-07T08:26:30+01:00" level=debug msg="Added outgoing tracing middleware dashboard@internal" entryPointName=traefik routerName=dashboard@internal middlewareName=tracing middlewareType=TracingForwarder
time="2023-11-07T08:26:30+01:00" level=debug msg="Creating middleware" middlewareType=StripPrefix entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal
time="2023-11-07T08:26:30+01:00" level=debug msg="Adding tracing to middleware" entryPointName=traefik middlewareName=dashboard_stripprefix@internal routerName=dashboard@internal
time="2023-11-07T08:26:30+01:00" level=debug msg="Creating middleware" middlewareType=RedirectRegex entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal
time="2023-11-07T08:26:30+01:00" level=debug msg="Setting up redirection from ^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$ to ${1}/dashboard/" middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex entryPointName=traefik routerName=dashboard@internal
time="2023-11-07T08:26:30+01:00" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal
time="2023-11-07T08:26:30+01:00" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=traefik
time="2023-11-07T08:26:30+01:00" level=debug msg="Provider connection established with docker 23.0.1 (API 1.42)" providerName=docker

time="2023-11-07T08:26:30+01:00" level=debug msg="Filtering disabled container" providerName=docker container=portainer-agent-9g6c03ga7kumgy77lqcs4cbqp
time="2023-11-07T08:26:30+01:00" level=debug msg="Filtering disabled container" providerName=docker container=portainer-agent-rpwpqn4hz3hrnv0dn33dxt6b1

time="2023-11-07T08:26:30+01:00" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"ansible\":{\"entryPoints\":[\"websecure\"],\"service\":\"ansible\",\"rule\":\"Host(`ansible.mycompany.com`)\",\"tls\":{}},\"api\":{\"middlewares\":[\"auth\"],\"service\":\"api@internal\",\"rule\":\"Host(`traefik.mycompany.com`)\"},\"docs\":{\"entryPoints\":[\"websecure\"],\"service\":\"docs\",\"rule\":\"Host(`docs.mycompany.com`)\",\"tls\":{}},\"portainer\":{\"entryPoints\":[\"web\"],\"service\":\"portainer\",\"rule\":\"Host(`portainer.mycompany.com`)\"},\"app1\":{\"entryPoints\":[\"websecure\"],\"service\":\"app1\",\"rule\":\"Host(`app1.mycompany.com`)\",\"tls\":{}},\"app2\":{\"entryPoints\":[\"websecure\"],\"service\":\"app2\",\"rule\":\"Host(`app2.mycompany.com`)\",\"tls\":{}}},\"services\":{\"ansible\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http:\/\/10.0.1.54:3000\"}],\"passHostHeader\":true}},\"docs\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http:\/\/10.0.1.227:80\"}],\"passHostHeader\":true}},\"portainer\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http:\/\/10.0.1.3:9000\"}],\"passHostHeader\":true}},\"app1\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http:\/\/10.0.1.236:80\"}],\"passHostHeader\":true}},\"app2\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http:\/\/10.0.1.94:80\"}],\"passHostHeader\":true}},\"traefik\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http:\/\/10.0.1.221:8080\"}],\"passHostHeader\":true}}},\"middlewares\":{\"auth\":{\"basicAuth\":{\"users\":[\"traefik:$2y$10$9UwPv.k1DAj\/3wjtT66TGetv9OJXStBGmLLGTPE9Ix30vsnckoae2\"]}}}},\"tcp\":{},\"udp\":{}}" providerName=docker
time="2023-11-07T08:26:30+01:00" level=debug msg="No entryPoint defined for this router, using the default one(s) instead: [mqttsecure web websecure]" routerName=api
time="2023-11-07T08:26:30+01:00" level=debug msg="No store is defined to add the certificate MIIIgjCCBmqgAwIBAgITRQADzZmlO57mTGHzYgAAAAPNmTANBg, it will be added to the default store."
time="2023-11-07T08:26:30+01:00" level=debug msg="Adding certificate for domain(s) *.mycompany.com,docker.mycompany.com,server123.mycompany.com"

time="2023-11-07T08:26:30+01:00" level=debug msg="Creating middleware" routerName=portainer@docker serviceName=portainer middlewareName=pipelining middlewareType=Pipelining entryPointName=web
time="2023-11-07T08:26:30+01:00" level=debug msg="Creating load-balancer" entryPointName=web routerName=portainer@docker serviceName=portainer
time="2023-11-07T08:26:30+01:00" level=debug msg="Creating server 0 http://10.0.1.3:9000" routerName=portainer@docker serviceName=portainer entryPointName=web serverName=0
time="2023-11-07T08:26:30+01:00" level=debug msg="child http://10.0.1.3:9000 now UP"
time="2023-11-07T08:26:30+01:00" level=debug msg="Propagating new UP status"
time="2023-11-07T08:26:30+01:00" level=debug msg="Added outgoing tracing middleware portainer" middlewareName=tracing middlewareType=TracingForwarder entryPointName=web routerName=portainer@docker

time="2023-11-07T08:26:30+01:00" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2023-11-07T08:26:30+01:00" level=debug msg="Added outgoing tracing middleware api@internal" middlewareType=TracingForwarder routerName=api@internal entryPointName=traefik middlewareName=tracing
time="2023-11-07T08:26:30+01:00" level=debug msg="Added outgoing tracing middleware dashboard@internal" routerName=dashboard@internal middlewareName=tracing middlewareType=TracingForwarder entryPointName=traefik
time="2023-11-07T08:26:30+01:00" level=debug msg="Creating middleware" routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix entryPointName=traefik
time="2023-11-07T08:26:30+01:00" level=debug msg="Adding tracing to middleware" routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal entryPointName=traefik
time="2023-11-07T08:26:30+01:00" level=debug msg="Creating middleware" middlewareType=RedirectRegex middlewareName=dashboard_redirect@internal entryPointName=traefik routerName=dashboard@internal
time="2023-11-07T08:26:30+01:00" level=debug msg="Setting up redirection from ^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$ to ${1}/dashboard/" entryPointName=traefik routerName=dashboard@internal middlewareType=RedirectRegex middlewareName=dashboard_redirect@internal
time="2023-11-07T08:26:30+01:00" level=debug msg="Adding tracing to middleware" middlewareName=dashboard_redirect@internal entryPointName=traefik routerName=dashboard@internal
time="2023-11-07T08:26:30+01:00" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=traefik middlewareName=traefik-internal-recovery
time="2023-11-07T08:26:30+01:00" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=mqttsecure middlewareName=traefik-internal-recovery
time="2023-11-07T08:26:30+01:00" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery

time="2023-11-07T08:26:30+01:00" level=debug msg="Creating middleware" serviceName=ansible middlewareName=pipelining middlewareType=Pipelining entryPointName=websecure routerName=ansible@docker
time="2023-11-07T08:26:30+01:00" level=debug msg="Creating load-balancer" routerName=ansible@docker serviceName=ansible entryPointName=websecure
time="2023-11-07T08:26:30+01:00" level=debug msg="Creating server 0 http://10.0.1.54:3000" entryPointName=websecure serverName=0 routerName=ansible@docker serviceName=ansible
time="2023-11-07T08:26:30+01:00" level=debug msg="child http://10.0.1.54:3000 now UP"
time="2023-11-07T08:26:30+01:00" level=debug msg="Propagating new UP status"
time="2023-11-07T08:26:30+01:00" level=debug msg="Added outgoing tracing middleware ansible" routerName=ansible@docker middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure

time="2023-11-07T08:26:30+01:00" level=debug msg="Creating middleware" middlewareName=pipelining middlewareType=Pipelining entryPointName=websecure routerName=docs@docker serviceName=docs
time="2023-11-07T08:26:30+01:00" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=docs@docker serviceName=docs
time="2023-11-07T08:26:30+01:00" level=debug msg="Creating server 0 http://10.0.1.227:80" routerName=docs@docker serviceName=docs serverName=0 entryPointName=websecure
time="2023-11-07T08:26:30+01:00" level=debug msg="child http://10.0.1.227:80 now UP"
time="2023-11-07T08:26:30+01:00" level=debug msg="Propagating new UP status"
time="2023-11-07T08:26:30+01:00" level=debug msg="Added outgoing tracing middleware docs" middlewareType=TracingForwarder routerName=docs@docker entryPointName=websecure middlewareName=tracing

time="2023-11-07T08:26:30+01:00" level=debug msg="Creating middleware" routerName=app2@docker serviceName=app2 middlewareName=pipelining middlewareType=Pipelining entryPointName=websecure
time="2023-11-07T08:26:30+01:00" level=debug msg="Creating load-balancer" routerName=app2@docker serviceName=app2 entryPointName=websecure
time="2023-11-07T08:26:30+01:00" level=debug msg="Creating server 0 http://10.0.1.94:80" routerName=app2@docker serviceName=app2 serverName=0 entryPointName=websecure
time="2023-11-07T08:26:30+01:00" level=debug msg="child http://10.0.1.94:80 now UP"
time="2023-11-07T08:26:30+01:00" level=debug msg="Propagating new UP status"
time="2023-11-07T08:26:30+01:00" level=debug msg="Added outgoing tracing middleware app2" routerName=app2@docker middlewareType=TracingForwarder middlewareName=tracing entryPointName=websecure

time="2023-11-07T08:26:30+01:00" level=debug msg="Creating middleware" serviceName=app1 middlewareName=pipelining middlewareType=Pipelining entryPointName=websecure routerName=app1@docker
time="2023-11-07T08:26:30+01:00" level=debug msg="Creating load-balancer" routerName=app1@docker serviceName=app1 entryPointName=websecure
time="2023-11-07T08:26:30+01:00" level=debug msg="Creating server 0 http://10.0.1.236:80" entryPointName=websecure routerName=app1@docker serviceName=app1 serverName=0
time="2023-11-07T08:26:30+01:00" level=debug msg="child http://10.0.1.236:80 now UP"
time="2023-11-07T08:26:30+01:00" level=debug msg="Propagating new UP status"
time="2023-11-07T08:26:30+01:00" level=debug msg="Added outgoing tracing middleware app1" middlewareType=TracingForwarder entryPointName=websecure routerName=app1@docker middlewareName=tracing

time="2023-11-07T08:26:30+01:00" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=websecure middlewareName=traefik-internal-recovery
time="2023-11-07T08:26:30+01:00" level=debug msg="Adding route for ansible.mycompany.com with TLS options default" entryPointName=websecure
time="2023-11-07T08:26:30+01:00" level=debug msg="Adding route for docs.mycompany.com with TLS options default" entryPointName=websecure
time="2023-11-07T08:26:30+01:00" level=debug msg="Adding route for app1.mycompany.com with TLS options default" entryPointName=websecure
time="2023-11-07T08:26:30+01:00" level=debug msg="Adding route for app2.mycompany.com with TLS options default" entryPointName=websecure
time="2023-11-07T08:26:31+01:00" level=debug msg="Authentication succeeded" middlewareName=auth@docker middlewareType=BasicAuth
time="2023-11-07T08:26:33+01:00" level=debug msg="Authentication succeeded" middlewareName=auth@docker middlewareType=BasicAuth
time="2023-11-07T08:26:34+01:00" level=debug msg="Authentication succeeded" middlewareName=auth@docker middlewareType=BasicAuth
time="2023-11-07T08:26:38+01:00" level=debug msg="Authentication succeeded" middlewareName=auth@docker middlewareType=BasicAuth
time="2023-11-07T08:26:39+01:00" level=debug msg="Authentication succeeded" middlewareType=BasicAuth middlewareName=auth@docker
time="2023-11-07T08:26:42+01:00" level=debug msg="'500 Internal Server Error' caused by: tls: failed to verify certificate: x509: cannot validate certificate for 10.0.1.236 because it doesn't contain any IP SANs"