Kubernetes: servers transport not found

BernhardBerbuir · December 6, 2021, 4:57pm

I want to use a self-signed certificate at a Pod. Therefore, traefik must be informed about the RootCA which signed the certificate. I'm using only a root CA, no intermediate CA.

I have created the following Kubernetes resources at the namespace infra-logging:

a Secret elastic-certificate-chain with key tls.ca and a self signed certificate as value
a ServersTransport elasticsearch which references the Secret:

    apiVersion: traefik.containo.us/v1alpha1
    kind: ServersTransport
    metadata:
      name: elasticsearch
      namespace: infra-logging
    spec:
      rootCAsSecrets:
        - elastic-certificate-chain

a Service with the annotation traefik.ingress.kubernetes.io/service.serverstransport: elasticsearch@kubernetescrd

I get the following error message: servers transport not found elasticsearch@kubernetes. Including the namespace into the annotation (i.e. using infra-logging-elasticsearch@kubernetescrd) does not help (it only changes the error message to servers transport not found infra-logging-elasticsearch@kubernetescrd ).

I'm using traefik version version 2.5.4

DEBUG output:

Configuration received from provider kubernetes:

{
  "http": {
    "routers": {
      "elasticsearch-local-master-infra-logging-elasticsearch-local-master-intern-localhost": {
        "service": "infra-logging-elasticsearch-local-master-9200",
        "rule": "Host(`elasticsearch-local-master.intern.localhost`) && PathPrefix(`/`)"
      }
      ...
    },
    "services": {
      "infra-logging-elasticsearch-local-master-9200": {
        "loadBalancer": {
          "servers": [
            {
              "url": "https://172.26.1.5:9200"
            },
            {
              "url": "https://172.26.2.6:9200"
            },
            {
              "url": "https://172.26.3.6:9200"
            }
          ],
          "passHostHeader": true,
          "serversTransport": "elasticsearch@kubernetescrd"
        }
      },
      ...
      }
    }
  },
  "tcp": {}
}

{
  "entryPointName": "intern-web",
  "level": "error",
  "msg": "servers transport not found elasticsearch@kubernetescrd",
  "routerName": "elasticsearch-local-master-infra-logging-elasticsearch-local-master-intern-localhost@kubernetes",
  "time": "2021-12-06T16:17:56Z"
}
{
  "entryPointName": "intern-secure",
  "level": "debug",
  "msg": "Adding route for elasticsearch-local-master.intern.localhost with TLS options default",
  "time": "2021-12-06T16:17:56Z"
}

My GitHub issue was was automatically closed, so that I hope to find here somebody who has already used the ServersTransport feature.

jakubhajek · December 6, 2021, 10:49pm

Hello @BernhardBerbuir

Thanks for using Traefik and asking the question.

Can you please update the annotation to the following:

traefik.ingress.kubernetes.io/service.serverstransport: infra-logging-elasticsearch@kubernetescrd

https://doc.traefik.io/traefik/routing/providers/kubernetes-ingress/#annotations

BernhardBerbuir · December 7, 2021, 9:29am

Hi @jakubhajek,
I already tried to use the full qualified name:

but this does not work either

I assume that the same problem occurs like when I have used Middlewares: there is a subtle error in my ServersTransport configurtion and therefore the CRD is ignored silently by traefik.

Is it allowed to set only "rootCAsSecrets" or are other attributes mandatory?

jakubhajek · December 7, 2021, 1:08pm

Hey @BernhardBerbuir

Would you please share your configuration in order to reproduce the issue locally?

Thank you,

BernhardBerbuir · December 8, 2021, 9:46am

@jakubhajek I have create a mininal example for reproducing the problem:

---
# REMARK: There are two traefik instances deployed
# => annotations (K8S resources) and labels (traefik CRDs) are required
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/protocol: https
    kubernetes.io/ingress.class: ingress-intern
  name: elasticsearch-master
  namespace: default
spec:
  rules:
    - host: elasticsearch-master.intern.localhost
      http:
        paths:
          - backend:
              serviceName: elasticsearch-master
              servicePort: 9200
            path: /
            pathType: ImplementationSpecific
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    traefik.ingress.kubernetes.io/service.serverstransport: elasticsearch-master@kubernetescrd
  name: elasticsearch-master
  namespace: default
spec:
  ports:
    - name: https
      port: 9200
      protocol: TCP
      targetPort: 9200
    - name: transport
      port: 9300
      protocol: TCP
      targetPort: 9300
  selector:
    app: elasticsearch-master
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: v1
data:
  tls.ca: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
kind: Secret
metadata:
   name: elasticsearch-certificate-chain
   namespace: default
type: Opaque
---
apiVersion: traefik.containo.us/v1alpha1
kind: ServersTransport
metadata:
  labels:
    kubernetes.io/ingress.class: ingress-intern
  name: elasticsearch-master
  namespace: default
spec:
  rootCAsSecrets:
    - elasticsearch-certificate-chain
---
# Dummy pod (required as traefik skips services without endpoint)
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: elasticsearch-master
  name: elasticsearch-master
spec:
  containers:
    - image: nginx:stable
      name: nginx

remark: I have two traefik deployments. The label should ensure that traefik "find" the ServersTransport.

These are my trefik commandline arguments:

    - --entryPoints.intern-secure.address=:8443/tcp
    - --entryPoints.intern-web.address=:8000/tcp
    - --entryPoints.traefik.address=:9000/tcp
    - --api.dashboard=true
    - --ping=true
    - --providers.kubernetescrd
    - --providers.kubernetesingress
    - --entrypoints.intern-secure.http.tls=true
    - --entrypoints.intern-web.http.redirections.entryPoint.to=:443
    - --entrypoints.intern-web.http.redirections.entryPoint.scheme=https
    - --log.format=json
    - --log.level=DEBUG
    - --accesslog=true
    - --accesslog.format=json
    - --accesslog.fields.defaultmode=keep
    - --accesslog.fields.names.StartLocal=drop
    - --accesslog.fields.headers.defaultmode=drop
    - --accesslog.fields.headers.names.Accept=keep
    - --accesslog.fields.headers.names.Accept-Encoding=keep
    - --accesslog.fields.headers.names.Accept-Language=keep
    - --accesslog.fields.headers.names.Accept-Ranges=keep
    - --accesslog.fields.headers.names.Access-Control-Allow-Origin=keep
    - --accesslog.fields.headers.names.Cache-Control=keep
    - --accesslog.fields.headers.names.Connection=keep
    - --accesslog.fields.headers.names.Content-Encoding=keep
    - --accesslog.fields.headers.names.Content-Length=keep
    - --accesslog.fields.headers.names.Content-Type=keep
    - --accesslog.fields.headers.names.Date=keep
    - --accesslog.fields.headers.names.Dnt=keep
    - --accesslog.fields.headers.names.Docker-Content-Digest=keep
    - --accesslog.fields.headers.names.Etag=keep
    - --accesslog.fields.headers.names.Expires=keep
    - --accesslog.fields.headers.names.Origin=keep
    - --accesslog.fields.headers.names.Pragma=keep
    - --accesslog.fields.headers.names.Referrer-Policy=keep
    - --accesslog.fields.headers.names.Sec-Ch-Ua=keep
    - --accesslog.fields.headers.names.Sec-Ch-Ua-Mobile=keep
    - --accesslog.fields.headers.names.Sec-Fetch-Dest=keep
    - --accesslog.fields.headers.names.Sec-Fetch-Mode=keep
    - --accesslog.fields.headers.names.Sec-Fetch-Site=keep
    - --accesslog.fields.headers.names.Strict-Transport-Security=keep
    - --accesslog.fields.headers.names.User-Agent=keep
    - --accesslog.fields.headers.names.Vary=keep
    - --accesslog.fields.headers.names.Www-Authenticate=keep
    - --accesslog.fields.headers.names.X-Content-Type-Options=keep
    - --accesslog.fields.headers.names.X-Forwarded-For=keep
    - --accesslog.fields.headers.names.X-Forwarded-Host=keep
    - --accesslog.fields.headers.names.X-Forwarded-Port=keep
    - --accesslog.fields.headers.names.X-Forwarded-Proto=keep
    - --accesslog.fields.headers.names.X-Forwarded-Server=keep
    - --accesslog.fields.headers.names.X-Frame-Options=keep
    - --accesslog.fields.headers.names.X-Real-Ip=keep
    - --accesslog.fields.headers.names.X-Request-Id=keep
    - --accesslog.fields.headers.names.X-Runtime=keep
    - --accesslog.fields.headers.names.X-Xss-Protection=keep
    - --pilot.dashboard=false
    - --api=true
    - --entrypoints.intern-secure.forwardedheaders.trustedips=127.0.0.1,172.16.0.0/12
    - --entrypoints.intern-secure.http.middlewares=infra-routing-default-chain-intern@kubernetescrd
    - --global.checknewversion=false
    - --metrics.prometheus.buckets=3.000000
    - --metrics.prometheus=true
    - --providers.kubernetescrd.allowcrossnamespace=false
    - --providers.kubernetescrd.allowexternalnameservices=true
    - --providers.kubernetesingress.allowexternalnameservices=true
    - --providers.kubernetescrd.labelselector=kubernetes.io/ingress.class=ingress-intern
    - --providers.kubernetesingress.ingressclass=ingress-intern

jakubhajek · December 10, 2021, 8:24pm

Hello @BernhardBerbuir

Thank you for sharing your configuration.

In order to achieve what you have described rootCA's has to be added in a static configuration, please see the following links:

rootCa
here is the full example how to mount secrets with CA's to Traefik deployment

Alternatively you can use insecure mode. By enabling that feature Traefik will trust the server certificate.

I also created the entire configuration so you can test it on your side. In my example Whoami application is acting as Elasticsearch. The application expose port 443 and certificate and key that has been issued by CA added to Traefik deployment.

The deployment for Whoami application, please note the annotation serverscheme that is added to the service.

---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: whoami-tls
  labels:
    app: whoami
    task: tls
spec:
  replicas: 1
  selector:
    matchLabels:
      app: whoami
      task: tls
  template:
    metadata:
      labels:
        app: whoami
        task: tls
    spec:
      containers:
        - name: whoami
          image: traefik/whoami
          ports:
            - containerPort: 443
          volumeMounts:
            - name: whoami-cert
              mountPath: /var/run/tls
          command:
            - "/whoami"
            - "--cert=/var/run/tls/tls.crt"
            - "--key=/var/run/tls/tls.key"
            - "--port=443"
      volumes:
        - name: whoami-cert
          secret:
            secretName: whoami-cert
---
apiVersion: v1
kind: Service
metadata:
  name: whoami-tls
  annotations:
    traefik.ingress.kubernetes.io/service.serversscheme: https
spec:
  ports:
    - name: https
      port: 443
  selector:
    app: whoami
    task: tls

Here is the Ingress object with added Traefik's annotation. You can use IngressClass to distinguish Traefik instances.

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
     traefik.ingress.kubernetes.io/router.tls: "true"
     traefik.ingress.kubernetes.io/router.entrypoints: websecure
  name: elasticsearch-master
spec:
  ingressClassName: traefik
  rules:
    - host: elasticsearch-master.127.0.0.1.nip.io
      http:
        paths:
          - path: / 
            pathType: Exact
            backend:
              service:
                name: whoami-tls
                port: 
                  number: 443

IngressClass:

---

apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  labels:
    app.kubernetes.io/instance: traefik
  name: traefik
spec:
  controller: traefik.io/ingress-controller

And here is the entire Traefik deployment;

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/instance: traefik
    app.kubernetes.io/name: traefik
  name: traefik
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/instance: traefik
      app.kubernetes.io/name: traefik
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: traefik
        app.kubernetes.io/name: traefik
    spec:
      containers:
      - args:
        - --entryPoints.web.address=:8000/tcp
        - --entryPoints.websecure.address=:8443/tcp
        - --entryPoints.traefik.address=:9000/tcp
        - --api=true
        - --api.dashboard=true
        - --api.insecure=true
        - --ping=true
        - --providers.kubernetescrd
        - --providers.kubernetescrd.allowCrossNamespace=true
        - --providers.kubernetesingress=true
        - --providers.kubernetesingress.ingressclass=traefik
        - --serversTransport.rootCAs=/certs/tls.ca
        - --serversTransport.insecureSkipVerify=true
        - --log.level=DEBUG
        image: traefik:2.5.4
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /ping
            port: 9000
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 2
        name: traefik
        ports:
        - containerPort: 8000
          name: web
          protocol: TCP
        - containerPort: 8443
          name: websecure
          protocol: TCP
        - containerPort: 9000
          name: traefik
          protocol: TCP
        readinessProbe:
          failureThreshold: 1
          httpGet:
            path: /ping
            port: 9000
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 2
        resources:
          limits:
            cpu: "1"
            memory: 1000Mi
          requests:
            cpu: 100m
            memory: 50Mi
        volumeMounts:
        - mountPath: /data
          name: storage-volume
        - mountPath: /certs
          name: ca
          readOnly: true
      serviceAccount: traefik-ingress-controller
      serviceAccountName: traefik-ingress-controller
      volumes:
      - emptyDir: {}
        name: storage-volume
      - name: ca
        secret:
          secretName: ca

I hope that helps.
Regards,

BernhardBerbuir · December 14, 2021, 5:19pm

As far as I can see the example does not use ServersTransport + Secret for providing the root ca. It uses a static configuration file (referenced by a command line argument) from a Secret for configuring traefik.

Is there a reason why the Kubernetes "way of working" is not used? By using a central configuration file the flexibility and independence of namespace administrators is lost.

The documentation describes how to use ServersTransport as a Kubernetes CRD: Traefik & Kubernetes - Kind: ServersTransport. Is this approach no longer working?

jakubhajek · December 16, 2021, 10:48am

Hello @BernhardBerbuir

In the example I shared adding RootCA is done though static configuration. In order to do that, a secret that contains CA's has to be added to the Traefik Deployment. Then certificates have to be referenced in the ServerTransport
Those certificates are reachable globally.

In that case, the solution is to use ServersTransport on a service level. In that case the CA's are applied from Kubernetes secret. Traefik Services Documentation - Traefik

That feature is used to implement mTLS - Routing Configuration for Traefik CRD - Traefik

If I correctly understand your use case the example from my previous post should be good enough to trust self signed certificates presented by Elasticsearch.

Please let us know,

catsem · December 20, 2021, 11:16am

@BernhardBerbuir I just struggeled with the same topic. Finally got it working by using the "fully qualified" reference in the services annotation.
traefik.ingress.kubernetes.io/service.serverstransport: <namespace>-<ServersTransport name>@kubernetescrd
Maybe the parser is buggy and has troubles with the hypen character in your namespace identifier. Could you try to use a "hypenless" namespace instead?
IMHO the documentation related to ServersTransport with dynamic config could need a little overhauling. This took me hours.

BernhardBerbuir · December 27, 2021, 4:25pm

I have reviewed my original setup: when using the "fully qualified" reference then it works fine (even in a namespace with a dash -). Additionally, my internal setup missed the label at the ServersTransport for being recognized by the correct traefik instance.

Unfortunately, neither the web nor the log file provides any information about ServersTransport (but Middleware resource configuration is quite good visualized at the web dashboard). I have had a certificate problem (the certificate only contained the host name from inside the cluster but not the external host name) and traefik did not displayed any error message about the host name mismatch.
Adding a serverName to the ServersTransport solved the problem.

system · December 30, 2021, 4:25pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
"servers transport not found" Traefik v2 docker	3	3857	August 23, 2021
TLS to backend with KubernetesCRD Traefik v2 kubernetes-crd	3	544	September 21, 2020
Self Signed certificate in Traefik 2.10, AKS Traefik v2 kubernetes-crd , kubernetes-ingress	9	1057	December 17, 2023
TLS with traefik and Kubernetes Traefik v2 kubernetes-crd	0	928	January 3, 2024
serverTransport not using TLS to backend service Traefik v2 kubernetes-crd , kubernetes-ingress	1	730	August 26, 2023

Kubernetes: servers transport not found

Related topics