Issues with traefik responding 404 for services behind it with TLS

Traefik Traefik v3 (latest)
, ,
1

Hi once again! I'm back with some more issues. I am trying to access a database behind traefik which is secured using self-signed certificates. CockroachDB has a tool to generate certificates and I use it to generate the needed certificates for my cluster to be secure. I wrote a little script to generate the certificates on demand, place them in a shared volume which every node and traefik have access to. I added every alternate name I could possibly use including traefik itself as thats what I would like to hit directly to load balance access to the nodes. I have two seperate compose files, one secure and one insecure. Using the insecure file I can access everything totally fine. I can curl from one of the nodes to the other (healthcheck) or I can access the dashboard and healthcheck api directly through traefik. As for the secure compose setup, I can curl from one node to the other using the node's domain name and I get an appropriate response back but when I try to access the nodes through traefik I get 404 response. I am new to using TLS and I would appreciate some reading material if you could suggest any that would allow me to understand what it is I am missing.

here are the relevant sections of my files

docker-compose.insecure.yaml:

services:
  roach1:
    image: cockroachdb/cockroach:v23.2.4
    volumes:
      - "roach1:/cockroach/cockroach-data"
    command: start --cluster-name=crabby-roach-insecure --logtostderr=WARNING --log-file-verbosity=INFO --insecure  --http-addr=0.0.0.0:8081 --cache=.25   --accept-sql-without-tls --advertise-addr=roach1_insecure:26257 --join=roach1_insecure:26257
    container_name: roach1_insecure
    hostname: roach1_insecure
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=roachnet"
      # admin interface 
      - "traefik.http.routers.roach_admin.rule=PathPrefix(`/admin/db/`) && Host(`traefik`)"
      - "traefik.http.routers.roach_admin.service=roach_admin_svc"
      - "traefik.http.services.roach_admin_svc.loadbalancer.server.port=8081"
      - "traefik.http.routers.roach_admin.entrypoints=roach_admin"
      - "traefik.http.middlewares.strip_prefix.stripprefix.prefixes=/admin/db"
      - "traefik.http.routers.roach_admin.middlewares=strip_prefix@docker"
      # Sql command interface 
      - "traefik.tcp.routers.roach_sql.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.roach_sql.service=roach_sql_svc"
      - "traefik.tcp.routers.roach_sql.tls=false"
      - "traefik.tcp.services.roach_sql_svc.loadbalancer.server.port=26257"
      - "traefik.tcp.routers.roach_sql.entrypoints=roachdb_sql"
    networks:
      - roachnet
 db-init:
    image: baxydocker/db-init:latest
    container_name: init
    hostname: roach-init
    command: "/tool"
    environment:
      - HEALTHCHECK=http://traefik:8081/admin/db/api/v2/health/?ready=1
      - COCKROACH_HOST=traefik
      - COCKROACH_PORT=26257
      - COCKROACH_USER=root
      - COCKROACH_INSECURE=true
      # uncomment the following line when first initializing a cluster
      # - COCKROACH_INIT=
      - DATABASE_NAME=crabby_userdb
      - DATABASE_PASSWORD=test
      - DATABASE_USER=crabby
    depends_on:
      - traefik
    networks:
      - roachnet 

  traefik:
    image: traefik:v3.0
    container_name: traefik
    hostname: traefik
    command:
      - "--api=true"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.roach_admin=true"
      - "--entrypoints.roachdb_sql=true"
      - "--entrypoints.web=true"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.roach_admin.address=:8081"
      - "--entrypoints.roachdb_sql.address=:26257"
      - "--log=true"
      - "--log.filepath=/traefik.insecure.log"
      - "--log.level=INFO"
      - "--accesslog=true"
      - "--accesslog.format=json"
      - "--accesslog.addinternals=true"
      - "--accesslog.filepath=/traefik.insecure.access.log"
    ports:
      - "69:80"
      - "8080:8080"
      - "8081:8081"
      - "26257:26257"
    labels:
      - "traefik.http.routers.dashboard.rule=Host(`traefik.localhost`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
      - "traefik.http.routers.dashboard.service=api@internal"
    volumes:
      - "../../logs/traefik.insecure.log:/traefik.insecure.log"
      - "../../logs/traefik.insecure.access.log:/traefik.insecure.access.log"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    depends_on:
      - roach1
      - roach2
      - roach3
    networks:
      - roachnet

docker-compose.secure.yaml:

roach3:
    image: cockroachdb/cockroach:v23.2.4
    hostname: roach3.crabby-userdb.io
    volumes:
      - "roach3:/cockroach/cockroach-data"
      - "certs:/certs:ro"

    command: start --cluster-name=crabby-roach-secure --logtostderr=WARNING --log-file-verbosity=INFO  --http-addr=0.0.0.0:8080  --cache=.25    --advertise-addr=roach3.crabby-userdb.io:26257 --join=roach1.crabby-userdb.io:26257 --certs-dir=/certs
    container_name: roach3
    labels:

      - "traefik.enable=true"
      - "traefik.docker.network=roachnet"
      # Middleware for redirect
      # - "traefik.http.middlewares.db_mid.headers.ssltemporaryredirect=false"
      # - "traefik.http.middlewares.db_mid.headers.sslredirect=true"
      #Middleware for strip prefix 
      - "traefik.http.middlewares.strip.stripprefix.prefixes=/dev/db"
      # admin interface 
      - "traefik.http.routers.roach_admin.rule=Host(`traefik`)&&PathPrefix(`/dev/db/`)|| HostRegexp(`^.+\\.crabby-userdb\\.io$`) "
      - "traefik.http.routers.roach_admin.service=roach_admin_svc"
      - "traefik.http.services.roach_admin_svc.loadbalancer.server.port=8080"
      - "traefik.http.services.roach_admin_svc.loadbalancer.sticky=true"
      - "traefik.http.routers.roach_admin.tls=true"
      - "traefik.http.routers.roach_admin.entrypoints=traefik"
      - "traefik.http.routers.roach_admin.middlewares=strip@docker"
      # - "traefik.http.routers.roach_admin.middlewares=db_mid@docker"
      # - "traefik.http.routers.roach_admin.tls.options=default"
      # Sql command interface 
      - "traefik.tcp.routers.roach_sql.rule=HostSNIRegexp(`^.+\\.crabby-userdb\\.io$ || traefik`)"
      - "traefik.tcp.routers.roach_sql.service=roach_sql_svc"
      - "traefik.tcp.routers.roach_sql.tls.passthrough=true"
      - "traefik.tcp.services.roach_sql_svc.loadbalancer.server.port=26257"
      - "traefik.tcp.routers.roach_sql.entrypoints=roachdb_sql"
    # depends_on:
    #   - traefik
    networks:
      - roachnet
    depends_on:
      certs_gen_tool:
        condition: "service_completed_successfully"


  db-init:
    image: baxydocker/db-init:latest
    container_name: init
    hostname: roach-init
    command: "/tool"
    environment:
      - COCKROACH_HOST=traefik
      - COCKROACH_PORT=26257
      - COCKROACH_USER=root
      - COCKROACH_INSECURE=false
      - COCKROACH_CERTS_DIR=/certs
      # uncomment the following line when first initializing a cluster
      - COCKROACH_INIT=
      - DATABASE_NAME=crabby_userdb
      - DATABASE_PASSWORD=test
      - DATABASE_USER=crabby
      - HEALTHCHECK=https://traefik:8080/dev/db/api/v2/health/?ready=1
    depends_on:
      # - traefik
      certs_gen_tool:
        condition: "service_completed_successfully"
    networks:
      - roachnet
    volumes:
      - "certs:/certs"
  certs_gen_tool:
    image: baxydocker/certs_gen:latest
    container_name: certs_gen
    command: "/gen"
    environment:
      - CLIENT_USERNAME=crabby
      - NODE_ALTERNATIVE_NAMES=crabby-userdb.io *.crabby-userdb.io localhost traefik
      # - COCKROACH_SKIP_KEY_PERMISSION_CHECK=true
    volumes:
      - "certs:/.cockroach-certs"
    networks:
      - roachnet

  traefik:
    image: traefik:v3.0
    container_name: traefik
    hostname: traefik
    command:
      - "--api=true"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.watch=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.file=true"
      -  --providers.file.filename=dynamic.yml
      # - "--log.filepath=traefik.log"
      - "--entrypoints.roach_admin=true"
      - "--entrypoints.roachdb_sql=true"
      - "--entrypoints.web=true"
      - "--entrypoints.web.address=:80"
      # - "--entrypoints.roach_admin.address=:8080"
      - "--entrypoints.roachdb_sql.address=:26257"
      - "--log=true"
      - "--log.filepath=/traefik.log"
      - "--log.format=json"
      - "--log.level=INFO"
      - "--accesslog=true"
      - "--accesslog.format=json"
      - "--accesslog.addinternals=true"
      - "--accesslog.filepath=/traefik.access.log"
      # INFO: maybe useful
      # - "--serverstransport.rootcas=/certs/.cockroach-certs/ca.crt"
    ports:
      - "69:80"
      - "8080:8080"
      - "8081:8081"
      - "26257:26257"
    labels:
      - "traefik.http.routers.dashboard.rule=Host(`traefik.localhost`) && PathPrefix(`/dashboard`))"
      # - "traefik.http.routers.dashboard.service=api@internal"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "../../config/traefik/dynamic.yml:/dynamic.yml"
      - "../../logs/traefik.log:/traefik.log"
      - "../../logs/traefik.access.log:/traefik.access.log"
      - "certs:/certs:ro"
    depends_on:
      # roach1:
      #   condition: "service_healthy"
      # roach2:
      #   condition: "service_healthy"
      # roach3:
      #   condition: "service_healthy"
      certs_gen_tool:
        condition: "service_completed_successfully"
    networks:
      - roachnet
volumes:
  roach1:
  roach2:
  roach3:
  certs:

traefik.log:

{"level":"info","version":"3.0.0-rc5","time":"2024-06-19T19:06:56Z","message":"Traefik version 3.0.0-rc5 built on 2024-04-11T16:27:55Z"}
{"level":"info","time":"2024-06-19T19:06:56Z","message":"\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"}
{"level":"info","time":"2024-06-19T19:06:56Z","message":"Starting provider aggregator aggregator.ProviderAggregator"}
{"level":"info","time":"2024-06-19T19:06:56Z","message":"Starting provider *file.Provider"}
{"level":"info","time":"2024-06-19T19:06:56Z","message":"Starting provider *traefik.Provider"}
{"level":"info","time":"2024-06-19T19:06:56Z","message":"Starting provider *docker.Provider"}
{"level":"info","time":"2024-06-19T19:06:56Z","message":"Starting provider *acme.ChallengeTLSALPN"}
{"level":"error","tlsStoreName":"default","error":"unable to generate TLS certificate : tls: failed to find any PEM data in certificate input","time":"2024-06-19T19:06:56Z","message":"Unable to append certificate /certs/.cockroach-certs/node.crt to store"}

traefik.access.log:

{"ClientAddr":"172.19.0.2:58444","ClientHost":"172.19.0.2","ClientPort":"58444","ClientUsername":"-","DownstreamContentSize":79,"DownstreamStatus":307,"Duration":900303,"OriginContentSize":79,"OriginDuration":851727,"OriginStatus":307,"Overhead":48576,"RequestAddr":"traefik:8080","RequestContentSize":0,"RequestCount":139,"RequestHost":"traefik","RequestMethod":"GET","RequestPath":"/dev/db/api/v2/health/?ready=1","RequestPort":"8080","RequestProtocol":"HTTP/1.1","RequestScheme":"https","RetryAttempts":0,"RouterName":"roach_admin@docker","ServiceAddr":"172.19.0.4:8080","ServiceName":"roach_admin_svc@docker","ServiceURL":"http://172.19.0.4:8080","StartLocal":"2024-06-19T19:12:00.949150227Z","StartUTC":"2024-06-19T19:12:00.949150227Z","TLSCipher":"TLS_AES_128_GCM_SHA256","TLSVersion":"1.3","entryPointName":"traefik","level":"info","msg":"","time":"2024-06-19T19:12:00Z"}
{"ClientAddr":"172.19.0.2:58444","ClientHost":"172.19.0.2","ClientPort":"58444","ClientUsername":"-","DownstreamContentSize":19,"DownstreamStatus":404,"Duration":15147,"GzipRatio":0,"OriginContentSize":0,"OriginDuration":0,"OriginStatus":0,"Overhead":15147,"RequestAddr":"traefik:8080","RequestContentSize":0,"RequestCount":140,"RequestHost":"traefik","RequestMethod":"GET","RequestPath":"/api/v2/health/?ready=1","RequestPort":"8080","RequestProtocol":"HTTP/1.1","RequestScheme":"https","RetryAttempts":0,"StartLocal":"2024-06-19T19:12:00.950727901Z","StartUTC":"2024-06-19T19:12:00.950727901Z","TLSCipher":"TLS_AES_128_GCM_SHA256","TLSVersion":"1.3","entryPointName":"traefik","level":"info","msg":"","time":"2024-06-19T19:12:00Z"}

dynamic.yaml:

http:
  serversTransports:
    cockroach:
      insecureSkipVerify: true
      rootCAs: /certs/.cockroach-certs/ca.crt


tls:
#   stores:
#     default:
#       defaultCertificate:
#          certFile: /certs/.cockroach-certs/node.crt
#          keyFile: /certs/.cockroach-key/node.key
  certificates:
#     # - certFile: /certs/.cockroach-certs/ca.crt
#     #   keyFile: /certs/.cockroach-key/ca.key
    - certFile: /certs/.cockroach-certs/node.crt
      keyFile: /certs/.cockroach-key/node.key
      stores: 
      - default
#   options:
#     default:
#       sniStrict: true
#   rootCAs:
#     - /certs/.cockroach-certs/ca.crt
  #   - /certs/.cockroach-certs/node.crt
  #

If there are comments anywhere it is stuff that I have tried based off of other posts or questions I found online. Thank you for your time!

2

Use 3 backticks before and after code/config to make it readable.

3

Got it! Sorry I think I fixed it!

4

How do you create

Clean up and remove:

Do you run multiple roach containers? Not sure if they can all use the same router name.

The TLS cert you supply seems incorrect. Is there a domain included? This is how Traefik matches a loaded cert with the according request to a domain.

5

And upgrade Traefik version from 3.0.0-rc5.

6

I run 3 roach containers right now, I haven't noticed any issues in the logs regarding using the same name for the router, it works as expected for the insecure cluster.

I'm not sure what you mean by this, I can provide you with the certificate so you can check it out.

roachnet was created manually previously when i was testing stuff for my application, it's external

7

@bluepuma77 I have now realized what you meant by my certificates seemed incorrect (I think). Let me know if I got this wrong. So I added the self-signed certificate authority as a rootCA, but the certificate I was providing in the tls.certificates section were in fact incorrect, I hadn't understood that these were supposed to be certificates to be used in mTLS as client certificates for communication with a backend service. I updated these certs to be client certificates (the ones generated by the cockroachdb "cert" tool) and I changed the path to them as that was also inaccurate relative to where the certificates would be for the traefik container.

I have tried using curl from inside the docker network on one of the cockroachdb containers (roachnet which traefik and every container in the cluster has access to) and this is the verbose output:

*   Trying 172.19.0.6:8080...
* Connected to traefik (172.19.0.6) port 8080 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=TRAEFIK DEFAULT CERT
*  start date: Jun 25 22:15:58 2024 GMT
*  expire date: Jun 25 22:15:58 2025 GMT
*  issuer: CN=TRAEFIK DEFAULT CERT
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* Using Stream ID: 1 (easy handle 0x55bff2c530a0)
* TLSv1.2 (OUT), TLS header, Unknown (23):
> GET /dev/db/api/v2/health/?ready=1 HTTP/2
> Host: traefik:8080
> user-agent: curl/7.76.1
> accept: */*
>
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Unknown (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.2 (IN), TLS header, Unknown (23):
< HTTP/2 307
< content-type: text/html; charset=utf-8
< date: Tue, 25 Jun 2024 22:17:12 GMT
< location: https://traefik:8080/api/v2/health/?ready=1
< content-length: 79
<
* TLSv1.2 (IN), TLS header, Unknown (23):
* Ignoring the response-body
* Connection #0 to host traefik left intact
* Issue another request to this URL: 'https://traefik:8080/api/v2/health/?ready=1'
* Found bundle for host traefik: 0x55bff2c479f0 [can multiplex]
* Re-using existing connection! (#0) with host traefik
* Connected to traefik (172.19.0.6) port 8080 (#0)
* Using Stream ID: 3 (easy handle 0x55bff2c530a0)
* TLSv1.2 (OUT), TLS header, Unknown (23):
> GET /api/v2/health/?ready=1 HTTP/2
> Host: traefik:8080
> user-agent: curl/7.76.1
> accept: */*
>
* TLSv1.2 (IN), TLS header, Unknown (23):
< HTTP/2 404
< content-type: text/plain; charset=utf-8
< x-content-type-options: nosniff
< content-length: 19
< date: Tue, 25 Jun 2024 22:17:12 GMT
<
* TLSv1.2 (IN), TLS header, Unknown (23):
404 page not found
* Connection #0 to host traefik left intact

from what I can see, it seems like the request resolves to the load balancer properly but get redirected back to it after the prefix gets stripped so when the new request gets sent with the stripped prefix back to traefik, it returns 404 because I haven't setup a route for that specific path with 'traefik' as matching dns. Is there a way for me to redirect the request towards the specific container I want the trafic to go to because in the certs generated for the database nodes, one of the alternate names is 'traefik'. Is there a way to change the dns in the request with some kind of middleware? Thank you for your help.

edit: I also noticed Traefik is using it's own certificate for the tls handshake, could there be an issue in the resolution of which service to send the request to because traefik's main domain is 'traefik' while it is an alternative name for my service?

8

Does the TLS cert have a "common name" set for the domain? This is how Traefik finds the right loaded TLS cert to use.

9

the "ca.crt" thaat I'm trying to use as an root Certificate Authority has Common name "Cockroach CA" as it was generated by the cockroachdb "certs" tool. One of my server certificates, "node.crt", has the CN Node as it is a requirement from CockroachDB itself for its tls

Server certificate.

node.crt must be signed by ca.crt and must have CN=node and the list of IP addresses and DNS names listed in Subject Alternative Name field. CockroachDB also supports wildcard notation in DNS names.

Here is the full page