Https backend service

We have plenty of https -> http configurations operational but have been unabled to forward one HTTPS endpoint to a HTTPS backend service.
I read most of the relavant posts on this forum which all boil down to setting insecureSkipVerify to true however, that doesn't work for us, we're getting a stuborn 404 page not found which actually seems to come from the backend service itself.

Any idea's how to resolve this ?

We'd like to set up the backend service with a self-signed certificate.

Static configuration

api:
  dashboard: true
  insecure: true
  debug: true

entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
          permanent: true
  websecure:
    address: ":443"

providers:
  file:
    directory: "/etc/traefik/dynamic-config"
    watch: true

serversTransport:
  insecureSkipVerify: true

dynamic configuration - tls (works for the other endpoint - so must be correct)

tls:
  certificates:
    - certFile: /etc/traefik/bruyland.crt
      keyFile: /etc/traefik/bruyland.key

Dynamic config for this specific endpoint

http:
  routers:
    test-router:
      rule: "Host(`rp-test2.bruyland.be`) || Host(`rp-test2`)"
      entryPoints:
        - websecure
      service: test-service
      tls: {}
  services:
    test-service:
      loadBalancer:
        servers:
          - url: "https://uitpak2.bruyland.be/"
#          - url: "https://www.google.com/"

This is what I get in the access logging

192.168.2.47 - - [19/Jan/2023:09:44:52 +0000] "GET /api HTTP/2.0" 404 19 "-" "-" 462 "test-router@file" "https://uitpak2.bruyland.be/" 1ms
192.168.2.47 - - [19/Jan/2023:09:44:52 +0000] "GET /api HTTP/2.0" 404 19 "-" "-" 463 "test-router@file" "https://uitpak2.bruyland.be/" 1ms
192.168.2.47 - - [19/Jan/2023:09:44:52 +0000] "GET /api HTTP/2.0" 404 19 "-" "-" 464 "test-router@file" "https://uitpak2.bruyland.be/" 1ms
192.168.2.47 - - [19/Jan/2023:09:44:53 +0000] "GET /api HTTP/2.0" 404 19 "-" "-" 465 "test-router@file" "https://uitpak2.bruyland.be/" 1ms
192.168.2.47 - - [19/Jan/2023:09:44:53 +0000] "GET /api HTTP/2.0" 404 19 "-" "-" 466 "test-router@file" "https://uitpak2.bruyland.be/" 1ms

When I replace the backend service with the google page, I get a 404 repoly from google.

Does your service at uitpak2.bruyland.be have a working /api path?

The address www.google.com/api does not exist, so you will get a 404.

It does.
Don't know why the /api path is in the logs.
I changed to backend service to a whoami test service, same result whithout the /api paths

192.168.2.47 - - [19/Jan/2023:10:53:33 +0000] "GET / HTTP/2.0" 404 19 "-" "-" 481 "test-router@file" "https://whoami.bruyland.be/" 1ms
192.168.2.47 - - [19/Jan/2023:10:53:33 +0000] "GET / HTTP/2.0" 404 19 "-" "-" 482 "test-router@file" "https://whoami.bruyland.be/" 1ms
192.168.2.47 - - [19/Jan/2023:10:53:34 +0000] "GET / HTTP/2.0" 404 19 "-" "-" 483 "test-router@file" "https://whoami.bruyland.be/" 1ms
192.168.2.47 - - [19/Jan/2023:10:53:34 +0000] "GET / HTTP/2.0" 404 19 "-" "-" 484 "test-router@file" "https://whoami.bruyland.be/" 0ms

Are you using containers?

Go into your Traefik container and try a wget with your target:

docker exec -it <traefik-container> sh
wget https://whoami.bruyland.be

Yes traefik is running in a docker container. I get this error message

Connecting to whoami.bruyland.be (192.168.1.67:443)
ssl_client: whoami.bruyland.be: certificate verification failed: unable to get local issuer certificate
wget: error getting response: Connection reset by peer

There is a 404 in the browser with this in the access log

192.168.2.47 - - [23/Jan/2023:09:01:58 +0000] "GET / HTTP/2.0" 404 19 "-" "-" 10 "test-router@file" "https://whoami.bruyland.be" 5ms

Turns out there was an intermediate certificate missing on the host. I installed it now, and the error on the host has gone away.
Unfortunately the error message inside the traefik container doesn't go away. Does traefik not use the ca certificates of the host machine ?

I have tried concatenate the intermediate certificate to the bruyland.cert file - no change
I have added the clientAuth options to the tls dynamic config - still no change

tls:
  certificates:
    - certFile: /etc/traefik/bruyland.crt
      keyFile: /etc/traefik/bruyland.key
  options:
    default:
      sniStrict: false
      clientAuth:
        caFiles: /etc/traefik/digicert.crt
        clientAuthType: RequestClientCert