Portainer behind Traefik - Bad certificate - insecureSkipVerify: true does not kick in?

[Dal]

Hi.
I'm trying to set up Portainer with OAuth Authentication from Keycloak.
Both Portainer and Keycloak are behind Traefik.
But it does not work.
And the traefik log tells me:

level=debug msg="http: TLS handshake error from 192.168.96.1:36486: remote error: tls: bad certificate"

I found this post that should think resolved it but no:

Traefik portainer bad certificate

Because Portainer is sitting behind Traefik, it can not generate a validated and signed cert. It will always create a custom cert that Traefik will not accept. (yes, there may be exceptions, but why go the hard way?) Either you go along the official Portainer+Traefik docs with port 9000 or you use insecureskipverify. Be aware you can't simply use insecureskipverify on a service, but you must create and assign serversTransport. ## Dynamic configuration http: serversTransports: mytranspor…

In my traefik.yml file I have this entry:

serversTransport:
  insecureSkipVerify: true

And my portainer docker compose file has these labels:

labels:
      - traefik.enable=true
      - traefik.http.routers.portainer.rule=Host("portainer.domain.tld")
      - traefik.http.routers.portainer.entryPoints=https
      - traefik.http.routers.portainer.tls=true
      - traefik.http.services.portainer.loadbalancer.server.port=9000

So I don't get it.
It is like insecureSkipVerify: true does not kick in

Am I missing something?

Thanks

[bluepuma77]

From the IP I would say the TLS error happens between browser client and Traefik, insecureSkipVerify is used between Traefik and target service.

I don't think there are any complications with Portainer in general, here an example compose file for stack deploy with Docker Swarm:

version: '3.9'

services:
  agent:
    image: portainer/agent:2.18.2
    environment:
      - LOG_LEVEL=DEBUG
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/lib/docker/volumes:/var/lib/docker/volumes
    networks:
      - agent_network
    deploy:
      mode: global
      placement:
        constraints: [node.platform.os == linux]

  portainer:
    image: portainer/portainer-ce:2.18.2
    command: -H tcp://tasks.agent:9001 --tlsskipverify
    #ports:
    #  - "9443:9443"
    #  - "9000:9000"
    #  - "8000:8000"
    volumes:
      - portainer_data:/data
    networks:
      - agent_network
      - proxy
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.role == manager
          - node.hostname == server1
      labels:
        - "traefik.enable=true"
        - "traefik.docker.network=proxy"
        - "traefik.http.routers.portainer.entrypoints=websecure"
        - "traefik.http.routers.portainer.rule=Host(`portainer.example.com`)"
        - "traefik.http.services.portainer.loadbalancer.server.port=9000"
        - "traefik.http.services.portainer.loadbalancer.passhostheader=true"

networks:
  agent_network:
    driver: overlay
    driver_opts:
      com.docker.network.driver.mtu: 1400
    attachable: true
  proxy:
    external: true

volumes:
  portainer_data:

[Dal]

@bluepuma77

From the IP I would say the TLS error happens between browser client and Traefik, insecureSkipVerify is used between Traefik and target service.

It is not.
The 192.168.96-network is actually docker network ip range, as can be seen here:

docker network inspect elastic
[
    {
        "Name": "elastic",
        "Id": "3a4956aa945056c77f1517b45b3e7c015e6095a0958feca988482321a97beaf2",
        "Created": "2024-02-29T14:26:16.78983627+01:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "192.168.96.0/20",
                    "Gateway": "192.168.96.1"

[Dal]

I changed the network on all actual containers, because the 192.168.96 network didn't sit right with me, but it didn't help:

http: TLS handshake error from 172.18.0.1:51378: remote error: tls: bad certificate"

I also experienced the exact same error when trying to set up OpenID Connect in my Gitea docker container.

[bluepuma77]

Works for me:

services:
  portainer:
    image: portainer/portainer-ce:latest
    networks:
      - proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - portainer_data:/data
    labels:
      - traefik.enable=true
      - traefik.http.routers.portainer.rule=Host(`portainer.example.com`)
      - traefik.http.services.portainer.loadbalancer.server.port=9000

volumes:
  portainer_data:

networks:
  proxy:
    name: proxy

Share your full Traefik static and dynamic config, and docker-compose.yml if used.

[Dal]

Sure.
Here is the docker-compose.yml:

docker-compose.yml

version: '3'
services:
  traefik:
    container_name: traefik
    image: traefik:latest
    ports:
      - 80:80
      - 443:443
      - 8080:8080
    volumes:
      - ./data:/etc/traefik
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - frontend
    labels:
      - traefik.enable=true
      - traefik.http.routers.api.rule=Host("traefik.domain.tld")
      - traefik.http.routers.api.entryPoints=https
      - traefik.http.routers.api.service=api@internal
      - traefik.http.services.api.loadbalancer.server.port=8080
      - traefik.http.routers.api.tls=true
    restart: unless-stopped

networks:
  frontend:
    external: true

The static traefik.yml:

traefik.yml

global:
  checkNewVersion: true
  sendAnonymousUsage: false

serversTransport:
  insecureSkipVerify: true

entryPoints:
  http:
    address: :80
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https
      middlewares:
        - securityHeaders@file
  https:
    address: :443

providers:
  providersThrottleDuration: 2s
  file:
    directory: /etc/traefik/dynamic
    watch: true
  docker:
    watch: true
    network: frontend
    #defaultRule: "Host(`{{ index .Labels \"com.docker.compose.service\"}}.domain.tld`)"
    exposedByDefault: false

api:
  dashboard: true
  insecure: true

# Log level INFO|DEBUG|ERROR
log:
  level: DEBUG
  filePath: /etc/traefik/logfile.log
accessLog:
  filePath: /etc/traefik/access.log

The dynamic fileConfig.yml, located in the "dynamic" folder:

fileConfig.yml

http:
  middlewares:
    securityHeaders:
      headers:
        customResponseHeaders:
          X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex"
          X-Forwarded-Proto: "https"
          server: ""
        sslProxyHeaders:
          X-Forwarded-Proto: https
        referrerPolicy: "same-origin"
        hostsProxyHeaders:
          - "X-Forwarded-Host"
        customRequestHeaders:
          X-Forwarded-Proto: "https"
        contentTypeNosniff: true
        browserXssFilter: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsSeconds: 63072000
        stsPreload: true
    gzip:
      compress: {}

I also have a certificates.yml file that points to my "static" certificates. This file is also located in the "dynamic" folder:

certificates.yml

tls:
  certificates:
    - certFile: /etc/traefik/certificates/domain.tld.pem
      keyFile: /etc/traefik/certificates/domain.tld.key
      stores:
        - default
  stores:
    default:
      defaultCertificate:
        certFile: /etc/traefik/certificates/domain.tld.pem
        keyFile: /etc/traefik/certificates/domain.tld.key

[bluepuma77]

Can you add the portainer compose?

[Dal]

portainer_docker-compose.yml

version: '3.9'
name: portainer
services:
  portainer:
    container_name: portainer
    image: portainer/portainer-ce:latest
    #ports:
      #- "9000:9000"
      #- "9443:9443"
    restart: unless-stopped
    labels:
      - traefik.enable=true
      - traefik.http.routers.portainer.rule=Host("portainer.domain.tld")
      - traefik.http.routers.portainer.entryPoints=https
      - traefik.http.routers.portainer.tls=true
      - traefik.http.services.portainer.loadbalancer.server.port=9000
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./data:/data
    networks:
      - frontend

networks:
  frontend:
    external: true

[bluepuma77]

Portainer port 9000 should be http by default, no TLS. So I stand by my previous statement, that’s it’s probably about TLS between browser client and Traefik.

[Dal]

If this is the case it would undermine the point of using Traefik at all.
Traefik is supposed to proxy trafic, yes?
In other words, the browser client would never reach Portainer directly.
Only Traefik talks to the Portainer instance.

But as I said, I met this problem when trying to set up OIDC on Gitea as well.
There I was able to work around it by setting up the well-known config by calling the container name directly, like this

http://keycloak:8080/realms/realmname/.well-known/openid-configuration

Isn't this the job Traefik is supposed to do for me?
Resolving my https request to https://keycloak.domain.tld into http://keycloak:8080 ?

[bluepuma77]

It depends on how you want to run your setup. Should Traefik completely terminate TLS and forward (proxy) requests with plain http or do you want to use TLS internally, too?

If you use TLS internally, the question is if you can use the same cert for Traefik and target service for a domain, so you can share the cert between them. Or you use different TLS certs, then the internal one is usually a "default" one, so not LE, so you need Traefik to trust it (insekureSkipVerify or load it).

Both Traefik and target service can use a valid LE cert when you use tlsChallenge for Traefik and httpChallenge for target service, but then you need to enable the forwarding of well-known. Or you can mix with more complicated dnsChallenge.

[Dal]

@bluepuma77

It depends on how you want to run your setup. Should Traefik completely terminate TLS and forward (proxy) requests with plain http or do you want to use TLS internally, too?

This is what I want. I thought this is what I had done with my setup.
Are there some parameters I am missing in my configs?

@bluepuma77

If you use TLS internally, the question is if you can use the same cert for Traefik and target service for a domain, so you can share the cert between them

Use TLS internally would be nice, but setup sounds like a nightmare.

But it would be fantastic with a shared keystore for all containers.

Doesn't the Elastic stack require encryption now even between their containers now for example?

[bluepuma77]

I asked an "or" question and you respond:

@Dal

This is what I want.

Which one?

[Dal]

@bluepuma77

Should Traefik completely terminate TLS and forward (proxy) requests with plain http

This. This whole thread has been about this

[bluepuma77]

This will not work, as you need to use backticks in Host():

@Dal

traefik.http.routers.api.rule=Host("traefik.domain.tld")

With tls=true you enable custom loaded certs on the routers. Those should be paid TLS certs, domain name needs to be correct, and cert file needs 3 sections inside, key file needs one section inside. Check Traefik debug log if they are loaded.

[Dal]

I'm sorry but you lost me here.
cert file 3 sections?

I tried removing the tls=true line, but then I only get 404 page not found

[bluepuma77]

Check that your cert files are valid, check the content with cat or a text editor.

[Dal]

Sigh, it seems to be loaded.
This pops up in the traefik log file when Portainer is restarted:

time="2024-03-16T09:25:13Z" level=debug msg="Provider event received {Status:die ID:10b0f8437822c423b2fdc2acd32941d5e9f4e2f585d20f8d6734be1677b42357 From:portainer/portainer-ce:latest Type:container Action:die Actor:{ID:10b0f8437822c423b2fdc2acd32941d5e9f4e2f585d20f8d6734be1677b42357 Attributes:map[com.docker.compose.config-hash:ed86e4212ea09f77944071603d1bfb6ee416e8bab05603cf7a9c6d058437460a com.docker.compose.container-number:1 com.docker.compose.depends_on: com.docker.compose.image:sha256:1a0fb356ea35830bad2a93aea5a72c88385b3505490cf035a575122bbe094a81 com.docker.compose.oneoff:False com.docker.compose.project:portainer com.docker.compose.project.config_files:/home/dockeruser/docker/portainer/docker-compose.yml com.docker.compose.project.working_dir:/home/dockeruser/docker/portainer com.docker.compose.service:portainer com.docker.compose.version:2.24.7 com.docker.desktop.extension.api.version:>= 0.2.2 com.docker.desktop.extension.icon:https://portainer-io-assets.sfo2.cdn.digitaloceanspaces.com/logos/portainer.png com.docker.extension.additional-urls:[{\"title\":\"Website\",\"url\":\"https://www.portainer.io?utm_campaign=DockerCon&utm_source=DockerDesktop\"},{\"title\":\"Documentation\",\"url\":\"https://docs.portainer.io\"},{\"title\":\"Support\",\"url\":\"https://join.slack.com/t/portainer/shared_invite/zt-txh3ljab-52QHTyjCqbe5RibC2lcjKA\"}] com.docker.extension.detailed-description:<p data-renderer-start-pos=\"226\">Portainer&rsquo;s Docker Desktop extension gives you access to all of Portainer&rsquo;s rich management functionality within your docker desktop experience.</p><h2 data-renderer-start-pos=\"374\">With Portainer you can:</h2><ul><li>See all your running containers</li><li>Easily view all of your container logs</li><li>Console into containers</li><li>Easily deploy your code into containers using a simple form</li><li>Turn your YAML into custom templates for easy reuse</li></ul><h2 data-renderer-start-pos=\"660\">About Portainer&nbsp;</h2><p data-renderer-start-pos=\"680\">Portainer is the worlds&rsquo; most popular universal container management platform with more than 650,000 active monthly users. Portainer can be used to manage Docker Standalone, Kubernetes, Docker Swarm and Nomad environments through a single common interface. It includes a simple GitOps automation engine and a Kube API.&nbsp;</p><p data-renderer-start-pos=\"1006\">Portainer Business Edition is our fully supported commercial grade product for business-wide use. It includes all the functionality that businesses need to manage containers at scale. Visit <a class=\"sc-jKJlTe dPfAtb\" href=\"http://portainer.io/\" title=\"http://Portainer.io\" data-renderer-mark=\"true\">Portainer.io</a> to learn more about Portainer Business and <a class=\"sc-jKJlTe dPfAtb\" href=\"http://portainer.io/take-3?utm_campaign=DockerCon&amp;utm_source=Docker%20Desktop\" title=\"http://portainer.io/take-3?utm_campaign=DockerCon&amp;utm_source=Docker%20Desktop\" data-renderer-mark=\"true\">get 3 free nodes.</a></p> com.docker.extension.publisher-url:https://www.portainer.io com.docker.extension.screenshots:[{\"alt\": \"screenshot one\", \"url\": \"https://portainer-io-assets.sfo2.digitaloceanspaces.com/screenshots/docker-extension-1.png\"},{\"alt\": \"screenshot two\", \"url\": \"https://portainer-io-assets.sfo2.digitaloceanspaces.com/screenshots/docker-extension-2.png\"},{\"alt\": \"screenshot three\", \"url\": \"https://portainer-io-assets.sfo2.digitaloceanspaces.com/screenshots/docker-extension-3.png\"},{\"alt\": \"screenshot four\", \"url\": \"https://portainer-io-assets.sfo2.digitaloceanspaces.com/screenshots/docker-extension-4.png\"},{\"alt\": \"screenshot five\", \"url\": \"https://portainer-io-assets.sfo2.digitaloceanspaces.com/screenshots/docker-extension-5.png\"},{\"alt\": \"screenshot six\", \"url\": \"https://portainer-io-assets.sfo2.digitaloceanspaces.com/screenshots/docker-extension-6.png\"},{\"alt\": \"screenshot seven\", \"url\": \"https://portainer-io-assets.sfo2.digitaloceanspaces.com/screenshots/docker-extension-7.png\"},{\"alt\": \"screenshot eight\", \"url\": \"https://portainer-io-assets.sfo2.digitaloceanspaces.com/screenshots/docker-extension-8.png\"},{\"alt\": \"screenshot nine\", \"url\": \"https://portainer-io-assets.sfo2.digitaloceanspaces.com/screenshots/docker-extension-9.png\"}] execDuration:33721 exitCode:2 image:portainer/portainer-ce:latest io.portainer.server:true name:portainer org.opencontainers.image.description:Docker container management made simple, with the world’s most popular GUI-based container management platform. org.opencontainers.image.title:Portainer org.opencontainers.image.vendor:Portainer.io traefik.enable:true traefik.http.routers.portainer.entryPoints:https traefik.http.routers.portainer.rule:Host(`portainer.domain.tld`) traefik.http.services.portainer.loadbalancer.server.port:9000]} Scope:local Time:1710581113 TimeNano:1710581113898055837}" providerName=docker
time="2024-03-16T09:25:13Z" level=debug msg="Filtering disabled container" providerName=docker container=postgres-postgres-c4c7b5ac182f30cbce4b76f5dc65770095bb4b2327d834b68d5d5a0d933adf8a
time="2024-03-16T09:25:13Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"adminer\":{\"entryPoints\":[\"https\"],\"service\":\"adminer-adminer\",\"rule\":\"Host(\\\"adminer.domain.tld\\\")\",\"tls\":{}},\"api\":{\"entryPoints\":[\"https\"],\"service\":\"api@internal\",\"rule\":\"Host(\\\"traefik.domain.tld\\\")\",\"tls\":{}},\"code-server\":{\"entryPoints\":[\"https\"],\"service\":\"code-server-code-server\",\"rule\":\"Host(\\\"cs.domain.tld\\\")\",\"tls\":{}},\"gitea\":{\"entryPoints\":[\"https\"],\"service\":\"gitea\",\"rule\":\"Host(\\\"gitea.domain.tld\\\")\",\"tls\":{}},\"homepage\":{\"entryPoints\":[\"https\"],\"service\":\"homepage-homepage\",\"rule\":\"Host(\\\"dashboard.domain.tld\\\")\",\"tls\":{}},\"keycloak\":{\"entryPoints\":[\"https\"],\"service\":\"keycloak-keycloak\",\"rule\":\"Host(\\\"kc.domain.tld\\\")\",\"tls\":{}},\"pgadmin\":{\"entryPoints\":[\"https\"],\"service\":\"pgadmin-pgadmin\",\"rule\":\"Host(\\\"pgadmin.domain.tld\\\")\",\"tls\":{}}},\"services\":{\"adminer-adminer\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.18.0.7:8080\"}],\"passHostHeader\":true}},\"api\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.18.0.4:8080\"}],\"passHostHeader\":true}},\"code-server-code-server\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.18.0.3:8443\"}],\"passHostHeader\":true}},\"gitea\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.18.0.8:3000\"}],\"passHostHeader\":true}},\"homepage-homepage\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.18.0.9:3000\"}],\"passHostHeader\":true}},\"keycloak-keycloak\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.18.0.6:8080\"}],\"passHostHeader\":true}},\"pgadmin-pgadmin\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.18.0.5:80\"}],\"passHostHeader\":true}}},\"middlewares\":{\"keycloak-https-redirect\":{\"redirectScheme\":{\"scheme\":\"https\"}}}},\"tcp\":{},\"udp\":{}}" providerName=docker
time="2024-03-16T09:25:13Z" level=debug msg="No store is defined to add the certificate Bag Attributes\n    localKeyID: F2 C7 EF 9F 0B 6F A3 B5 1E A5 95 DF C4 EE 3F 48 DA BB E9 90 \nsubject=CN = *.domain.tld\nissuer=C = US, ST = Arizona, L = Scottsdale, O = \"Starfield Technologies, Inc.\", OU = http://certs.starfieldtech.com/repository/, CN = Starfield Secure Certificate Authority - G2\n-----BEGIN CERTIFICATE-----<SNIP> \n-----END CERTIFICATE-----\n, it will be added to the default store."
time="2024-03-16T09:25:13Z" level=debug msg="Adding certificate for domain(s) *.domain.tld,domain.tld"
time="2024-03-16T09:25:14Z" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default

[Dal]

I think I have got one step further to solving this.
I tried setting up nginx proxy manager, and faced the same problems there.. until I added the CA file from the authority that created my certificate.

But my question is:
Where in the traefik configuration (dynamic or static) do I point to that CA file?
What is the configuration parameter called?

Thanks

[bluepuma77]

That’s what I am telling you: a regular TLS cert file should contain 3 parts ("-BEGIN CERTIFICATE-").

If you buy one and don’t get a .pem with 3 parts, then they usually provide a doc which files need to be copied together for a valid TLS cert.