Kubernetes TLS unsupported versions

What did you do?

  1. Copied all contents of the Traefik Kubernetes Let's Encrypt Guide to my Controller. (CRD, Services, Deployment, Ingress)
  2. Removed Let's Encrypt from Args.
  3. Created TLS Secret for Ingress Route.
  4. Applied TLS Secret to Ingress Route.

What did you expect to see?

Traefik creating the ingress route and applying the Secret.

What did you see instead?

Traefik creating TLS errors and skipping events.

Output of traefik version: (What version of Traefik are you using?)

Version:      2.2.1
Codename:     chevrotin
Go version:   go1.14.2
Built:        2020-04-29T18:02:09Z
OS/Arch:      linux/arm

What is your environment & configuration (arguments, toml, provider, platform, ...)?

crd.yaml

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutes.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRoute
    plural: ingressroutes
    singular: ingressroute
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: middlewares.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: Middleware
    plural: middlewares
    singular: middleware
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutetcps.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteTCP
    plural: ingressroutetcps
    singular: ingressroutetcp
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressrouteudps.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteUDP
    plural: ingressrouteudps
    singular: ingressrouteudp
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsoptions.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSOption
    plural: tlsoptions
    singular: tlsoption
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsstores.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSStore
    plural: tlsstores
    singular: tlsstore
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: traefikservices.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TraefikService
    plural: traefikservices
    singular: traefikservice
  scope: Namespaced

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller

rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - traefik.containo.us
    resources:
      - middlewares
      - ingressroutes
      - traefikservices
      - ingressroutetcps
      - ingressrouteudps
      - tlsoptions
      - tlsstores
    verbs:
      - get
      - list
      - watch

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller

roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
  - kind: ServiceAccount
    name: traefik-ingress-controller
    namespace: production

services.yaml

apiVersion: v1
kind: Service
metadata:
  name: traefik
  namespace: production
spec:
  type: NodePort
  ports:
    - protocol: TCP
      name: web
      port: 8000
      nodePort: 30080
    - protocol: TCP
      name: admin
      port: 8080
    - protocol: TCP
      name: websecure
      port: 4443
      nodePort: 30443
  selector:
    app: traefik

---
apiVersion: v1
kind: Service
metadata:
  name: whoami
  namespace: production

spec:
  ports:
    - protocol: TCP
      name: web
      port: 80
  selector:
    app: whoami-web

secret.yaml

apiVersion: v1
data:
  tls.crt:  base64 crt was here
  tls.key: base64 key was here
kind: Secret
metadata:
  creationTimestamp: null
  name: default-certificate
  namespace: production
type: kubernetes.io/tls

deployment.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: production
  name: traefik-ingress-controller

---
kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: production
  name: traefik
  labels:
    app: traefik

spec:
  replicas: 2
  selector:
    matchLabels:
      app: traefik
  template:
    metadata:
      labels:
        app: traefik
    spec:
      serviceAccountName: traefik-ingress-controller
      containers:
        - name: traefik
          image: traefik:v2.2
          args:
            - --api.insecure
            - --accesslog
            - --entrypoints.web.Address=:8000
            - --entrypoints.websecure.Address=:4443
            - --providers.kubernetescrd
            - --log.level=DEBUG
          ports:
            - name: web
              containerPort: 8000
            - name: websecure
              containerPort: 4443
            - name: admin
              containerPort: 8080

---
kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: production
  name: whoami
  labels:
    app: whoami

spec:
  replicas: 2
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
        - name: whoami
          image: containous/whoami
          ports:
            - name: whoami-web
              containerPort: 80

ingress.yaml

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: simpleingressroute
  namespace: production
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`request.whitebox.app`) && PathPrefix(`/`)
    kind: Rule
    services:
    - name: whoami
      port: 80

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: ingressroutetls
  namespace: production
spec:
  entryPoints:
    - websecure
  routes:
  - match: Host(`request.whitebox.app`) && PathPrefix(`/`)
    kind: Rule
    services:
    - name: whoami
      port: 80

  tls:
    secretName: default-certificate
    options:              
      name: traefik-options          
      namespace: production       

tls.yaml

apiVersion: traefik.containo.us/v1alpha1
kind: TLSOption
metadata:
  name: traefik-options
  namespace: production

spec:
  minVersion: VersionTLS12

If applicable, please paste the log output in DEBUG level (--log.level=DEBUG switch)

time="2020-07-16T19:57:26Z" level=info msg="Configuration loaded from flags."
time="2020-07-16T19:57:26Z" level=info msg="Traefik version 2.2.1 built on 2020-04-29T18:02:09Z"
time="2020-07-16T19:57:26Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"traefik\":{\"address\":\":8080\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{},\"http\":{}},\"web\":{\"address\":\":8000\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{},\"http\":{}},\"websecure\":{\"address\":\":4443\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{},\"http\":{}}},\"providers\":{\"providersThrottleDuration\":2000000000,\"kubernetesCRD\":{}},\"api\":{\"insecure\":true,\"dashboard\":true},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"},\"accessLog\":{\"format\":\"common\",\"filters\":{},\"fields\":{\"defaultMode\":\"keep\",\"headers\":{\"defaultMode\":\"drop\"}}}}"
time="2020-07-16T19:57:26Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/contributing/data-collection/\n"
time="2020-07-16T19:57:26Z" level=info msg="Starting provider aggregator.ProviderAggregator {}"
time="2020-07-16T19:57:26Z" level=debug msg="Start TCP Server" entryPointName=traefik
time="2020-07-16T19:57:26Z" level=debug msg="Start TCP Server" entryPointName=web
time="2020-07-16T19:57:26Z" level=debug msg="Start TCP Server" entryPointName=websecure
time="2020-07-16T19:57:26Z" level=info msg="Starting provider *traefik.Provider {}"
time="2020-07-16T19:57:26Z" level=info msg="Starting provider *crd.Provider {}"
time="2020-07-16T19:57:26Z" level=debug msg="Using label selector: \"\"" providerName=kubernetescrd
time="2020-07-16T19:57:26Z" level=info msg="label selector is: \"\"" providerName=kubernetescrd
time="2020-07-16T19:57:26Z" level=info msg="Creating in-cluster Provider client" providerName=kubernetescrd
time="2020-07-16T19:57:26Z" level=debug msg="Configuration received from provider internal: {\"http\":{\"routers\":{\"api\":{\"entryPoints\":[\"traefik\"],\"service\":\"api@internal\",\"rule\":\"PathPrefix(`/api`)\",\"priority\":2147483646},\"dashboard\":{\"entryPoints\":[\"traefik\"],\"middlewares\":[\"dashboard_redirect@internal\",\"dashboard_stripprefix@internal\"],\"service\":\"dashboard@internal\",\"rule\":\"PathPrefix(`/`)\",\"priority\":2147483645}},\"services\":{\"api\":{},\"dashboard\":{},\"noop\":{}},\"middlewares\":{\"dashboard_redirect\":{\"redirectRegex\":{\"regex\":\"^(http:\\\\/\\\\/[^:\\\\/]+(:\\\\d+)?)\\\\/$\",\"replacement\":\"${1}/dashboard/\",\"permanent\":true}},\"dashboard_stripprefix\":{\"stripPrefix\":{\"prefixes\":[\"/dashboard/\",\"/dashboard\"]}}}},\"tcp\":{},\"tls\":{}}" providerName=internal
time="2020-07-16T19:57:26Z" level=debug msg="Added outgoing tracing middleware api@internal" entryPointName=traefik routerName=api@internal middlewareName=tracing middlewareType=TracingForwarder
time="2020-07-16T19:57:26Z" level=debug msg="Added outgoing tracing middleware dashboard@internal" entryPointName=traefik routerName=dashboard@internal middlewareName=tracing middlewareType=TracingForwarder
time="2020-07-16T19:57:26Z" level=debug msg="Creating middleware" routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix entryPointName=traefik
time="2020-07-16T19:57:26Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal
time="2020-07-16T19:57:26Z" level=debug msg="Creating middleware" routerName=dashboard@internal entryPointName=traefik middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex
time="2020-07-16T19:57:26Z" level=debug msg="Setting up redirection from ^(http:\\/\\/[^:\\/]+(:\\d+)?)\\/$ to ${1}/dashboard/" middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex routerName=dashboard@internal entryPointName=traefik
time="2020-07-16T19:57:26Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal
time="2020-07-16T19:57:26Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2020-07-16T19:57:26Z" level=debug msg="No default certificate, generating one"
time="2020-07-16T19:57:26Z" level=error msg="subset not found for production/whoami" namespace=production providerName=kubernetescrd ingress=simpleingressroute
time="2020-07-16T19:57:26Z" level=error msg="subset not found for production/whoami" ingress=ingressroutetls namespace=production providerName=kubernetescrd
time="2020-07-16T19:57:26Z" level=debug msg="Configuration received from provider kubernetescrd: {\"http\":{},\"tcp\":{},\"udp\":{},\"tls\":{\"options\":{\"production-traefik-options\":{\"minVersion\":\"VersionTLS12\",\"clientAuth\":{}}}}}" providerName=kubernetescrd
time="2020-07-16T19:57:26Z" level=error msg="subset not found for production/whoami" namespace=production providerName=kubernetescrd ingress=simpleingressroute
time="2020-07-16T19:57:26Z" level=error msg="subset not found for production/whoami" ingress=ingressroutetls namespace=production providerName=kubernetescrd
time="2020-07-16T19:57:26Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-16T19:57:27Z" level=error msg="subset not found for production/whoami" ingress=simpleingressroute namespace=production providerName=kubernetescrd
time="2020-07-16T19:57:27Z" level=error msg="subset not found for production/whoami" providerName=kubernetescrd namespace=production ingress=ingressroutetls
time="2020-07-16T19:57:27Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-16T19:57:27Z" level=debug msg="Configuration received from provider kubernetescrd: {\"http\":{\"routers\":{\"production-ingressroutetls-7df7c494de9ca427e2bb\":{\"entryPoints\":[\"websecure\"],\"service\":\"production-ingressroutetls-7df7c494de9ca427e2bb\",\"rule\":\"Host(`request.whitebox.app`) \\u0026\\u0026 PathPrefix(`/`)\",\"tls\":{\"options\":\"production-traefik-options\"}},\"production-simpleingressroute-7df7c494de9ca427e2bb\":{\"entryPoints\":[\"web\"],\"service\":\"production-simpleingressroute-7df7c494de9ca427e2bb\",\"rule\":\"Host(`request.whitebox.app`) \\u0026\\u0026 PathPrefix(`/`)\"}},\"services\":{\"production-ingressroutetls-7df7c494de9ca427e2bb\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.40.0.2:80\"}],\"passHostHeader\":true}},\"production-simpleingressroute-7df7c494de9ca427e2bb\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.40.0.2:80\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"udp\":{},\"tls\":{\"options\":{\"production-traefik-options\":{\"minVersion\":\"VersionTLS12\",\"clientAuth\":{}}}}}" providerName=kubernetescrd
time="2020-07-16T19:57:28Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-16T19:57:29Z" level=debug msg="Configuration received from provider kubernetescrd: {\"http\":{\"routers\":{\"production-ingressroutetls-7df7c494de9ca427e2bb\":{\"entryPoints\":[\"websecure\"],\"service\":\"production-ingressroutetls-7df7c494de9ca427e2bb\",\"rule\":\"Host(`request.whitebox.app`) \\u0026\\u0026 PathPrefix(`/`)\",\"tls\":{\"options\":\"production-traefik-options\"}},\"production-simpleingressroute-7df7c494de9ca427e2bb\":{\"entryPoints\":[\"web\"],\"service\":\"production-simpleingressroute-7df7c494de9ca427e2bb\",\"rule\":\"Host(`request.whitebox.app`) \\u0026\\u0026 PathPrefix(`/`)\"}},\"services\":{\"production-ingressroutetls-7df7c494de9ca427e2bb\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.40.0.2:80\"},{\"url\":\"http://10.44.0.3:80\"}],\"passHostHeader\":true}},\"production-simpleingressroute-7df7c494de9ca427e2bb\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.40.0.2:80\"},{\"url\":\"http://10.44.0.3:80\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"udp\":{},\"tls\":{\"options\":{\"production-traefik-options\":{\"minVersion\":\"VersionTLS12\",\"clientAuth\":{}}}}}" providerName=kubernetescrd
time="2020-07-16T19:57:29Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-16T19:57:30Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-16T19:57:31Z" level=debug msg="No store is defined to add the certificate MIIEpDCCA4ygAwIBAgIUX8H4DzPqZYSKmY8ILLsf8Qv/lMQwDQ, it will be added to the default store."
time="2020-07-16T19:57:31Z" level=debug msg="Adding certificate for domain(s) cloudflare origin certificate,*.whitebox.app,whitebox.app"
time="2020-07-16T19:57:31Z" level=debug msg="No default certificate, generating one"
time="2020-07-16T19:57:31Z" level=debug msg="http: TLS handshake error from 10.44.0.0:47734: tls: client offered only unsupported versions: []"
time="2020-07-16T19:57:31Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-16T19:57:32Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-16T19:57:33Z" level=debug msg="http: TLS handshake error from 10.44.0.0:47738: tls: client offered only unsupported versions: []"
time="2020-07-16T19:57:34Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-16T19:57:34Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-16T19:57:36Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-16T19:57:36Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-16T19:57:36Z" level=debug msg="Added outgoing tracing middleware api@internal" entryPointName=traefik routerName=api@internal middlewareName=tracing middlewareType=TracingForwarder
time="2020-07-16T19:57:36Z" level=debug msg="Added outgoing tracing middleware dashboard@internal" middlewareType=TracingForwarder routerName=dashboard@internal entryPointName=traefik middlewareName=tracing
time="2020-07-16T19:57:36Z" level=debug msg="Creating middleware" routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix entryPointName=traefik
time="2020-07-16T19:57:36Z" level=debug msg="Adding tracing to middleware" middlewareName=dashboard_stripprefix@internal entryPointName=traefik routerName=dashboard@internal
time="2020-07-16T19:57:36Z" level=debug msg="Creating middleware" routerName=dashboard@internal middlewareType=RedirectRegex middlewareName=dashboard_redirect@internal entryPointName=traefik
time="2020-07-16T19:57:36Z" level=debug msg="Setting up redirection from ^(http:\\/\\/[^:\\/]+(:\\d+)?)\\/$ to ${1}/dashboard/" middlewareName=dashboard_redirect@internal entryPointName=traefik routerName=dashboard@internal middlewareType=RedirectRegex
time="2020-07-16T19:57:36Z" level=debug msg="Adding tracing to middleware" middlewareName=dashboard_redirect@internal entryPointName=traefik routerName=dashboard@internal
time="2020-07-16T19:57:36Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2020-07-16T19:57:36Z" level=debug msg="No store is defined to add the certificate MIIEpDCCA4ygAwIBAgIUX8H4DzPqZYSKmY8ILLsf8Qv/lMQwDQ, it will be added to the default store."
time="2020-07-16T19:57:36Z" level=debug msg="Adding certificate for domain(s) cloudflare origin certificate,*.whitebox.app,whitebox.app"
time="2020-07-16T19:57:36Z" level=debug msg="No default certificate, generating one"
time="2020-07-16T19:57:38Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-16T19:57:38Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-16T19:57:39Z" level=debug msg="http: TLS handshake error from 10.44.0.0:60666: tls: client offered only unsupported versions: []"
time="2020-07-16T19:57:40Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-16T19:57:40Z" level=debug msg="Creating middleware" serviceName=production-simpleingressroute-7df7c494de9ca427e2bb middlewareName=pipelining middlewareType=Pipelining entryPointName=web routerName=production-simpleingressroute-7df7c494de9ca427e2bb@kubernetescrd
time="2020-07-16T19:57:40Z" level=debug msg="Creating load-balancer" routerName=production-simpleingressroute-7df7c494de9ca427e2bb@kubernetescrd serviceName=production-simpleingressroute-7df7c494de9ca427e2bb entryPointName=web
time="2020-07-16T19:57:40Z" level=debug msg="Creating server 0 http://10.40.0.2:80" entryPointName=web routerName=production-simpleingressroute-7df7c494de9ca427e2bb@kubernetescrd serviceName=production-simpleingressroute-7df7c494de9ca427e2bb serverName=0
time="2020-07-16T19:57:40Z" level=debug msg="Added outgoing tracing middleware production-simpleingressroute-7df7c494de9ca427e2bb" middlewareType=TracingForwarder entryPointName=web routerName=production-simpleingressroute-7df7c494de9ca427e2bb@kubernetescrd middlewareName=tracing
time="2020-07-16T19:57:40Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=web middlewareName=traefik-internal-recovery
time="2020-07-16T19:57:40Z" level=debug msg="Added outgoing tracing middleware api@internal" middlewareType=TracingForwarder entryPointName=traefik routerName=api@internal middlewareName=tracing
time="2020-07-16T19:57:40Z" level=debug msg="Added outgoing tracing middleware dashboard@internal" entryPointName=traefik routerName=dashboard@internal middlewareType=TracingForwarder middlewareName=tracing
time="2020-07-16T19:57:40Z" level=debug msg="Creating middleware" routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix entryPointName=traefik
time="2020-07-16T19:57:40Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal
time="2020-07-16T19:57:40Z" level=debug msg="Creating middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex
time="2020-07-16T19:57:40Z" level=debug msg="Setting up redirection from ^(http:\\/\\/[^:\\/]+(:\\d+)?)\\/$ to ${1}/dashboard/" middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex entryPointName=traefik routerName=dashboard@internal
time="2020-07-16T19:57:40Z" level=debug msg="Adding tracing to middleware" middlewareName=dashboard_redirect@internal routerName=dashboard@internal entryPointName=traefik
time="2020-07-16T19:57:40Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2020-07-16T19:57:40Z" level=debug msg="Creating middleware" middlewareType=Pipelining entryPointName=websecure routerName=production-ingressroutetls-7df7c494de9ca427e2bb@kubernetescrd serviceName=production-ingressroutetls-7df7c494de9ca427e2bb middlewareName=pipelining
time="2020-07-16T19:57:40Z" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=production-ingressroutetls-7df7c494de9ca427e2bb@kubernetescrd serviceName=production-ingressroutetls-7df7c494de9ca427e2bb
time="2020-07-16T19:57:40Z" level=debug msg="Creating server 0 http://10.40.0.2:80" serverName=0 entryPointName=websecure routerName=production-ingressroutetls-7df7c494de9ca427e2bb@kubernetescrd serviceName=production-ingressroutetls-7df7c494de9ca427e2bb
time="2020-07-16T19:57:40Z" level=debug msg="Added outgoing tracing middleware production-ingressroutetls-7df7c494de9ca427e2bb" entryPointName=websecure routerName=production-ingressroutetls-7df7c494de9ca427e2bb@kubernetescrd middlewareName=tracing middlewareType=TracingForwarder
time="2020-07-16T19:57:40Z" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2020-07-16T19:57:40Z" level=debug msg="Adding route for request.whitebox.app with TLS options production-traefik-options" entryPointName=websecure
time="2020-07-16T19:57:40Z" level=debug msg="No store is defined to add the certificate MIIEpDCCA4ygAwIBAgIUX8H4DzPqZYSKmY8ILLsf8Qv/lMQwDQ, it will be added to the default store."
time="2020-07-16T19:57:40Z" level=debug msg="Adding certificate for domain(s) cloudflare origin certificate,*.whitebox.app,whitebox.app"
time="2020-07-16T19:57:40Z" level=debug msg="No default certificate, generating one"
time="2020-07-16T19:57:40Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-16T19:57:41Z" level=debug msg="http: TLS handshake error from 10.44.0.0:60672: tls: client offered only unsupported versions: []"
time="2020-07-16T19:57:42Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-16T19:57:42Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-16T19:57:43Z" level=debug msg="Creating middleware" routerName=production-simpleingressroute-7df7c494de9ca427e2bb@kubernetescrd serviceName=production-simpleingressroute-7df7c494de9ca427e2bb middlewareName=pipelining middlewareType=Pipelining entryPointName=web
time="2020-07-16T19:57:43Z" level=debug msg="Creating load-balancer" entryPointName=web routerName=production-simpleingressroute-7df7c494de9ca427e2bb@kubernetescrd serviceName=production-simpleingressroute-7df7c494de9ca427e2bb
time="2020-07-16T19:57:43Z" level=debug msg="Creating server 0 http://10.40.0.2:80" serverName=0 entryPointName=web routerName=production-simpleingressroute-7df7c494de9ca427e2bb@kubernetescrd serviceName=production-simpleingressroute-7df7c494de9ca427e2bb
time="2020-07-16T19:57:43Z" level=debug msg="Creating server 1 http://10.44.0.3:80" entryPointName=web routerName=production-simpleingressroute-7df7c494de9ca427e2bb@kubernetescrd serviceName=production-simpleingressroute-7df7c494de9ca427e2bb serverName=1
time="2020-07-16T19:57:43Z" level=debug msg="Added outgoing tracing middleware production-simpleingressroute-7df7c494de9ca427e2bb" entryPointName=web routerName=production-simpleingressroute-7df7c494de9ca427e2bb@kubernetescrd middlewareName=tracing middlewareType=TracingForwarder
time="2020-07-16T19:57:43Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2020-07-16T19:57:43Z" level=debug msg="Added outgoing tracing middleware api@internal" middlewareName=tracing middlewareType=TracingForwarder routerName=api@internal entryPointName=traefik
time="2020-07-16T19:57:43Z" level=debug msg="Added outgoing tracing middleware dashboard@internal" entryPointName=traefik routerName=dashboard@internal middlewareName=tracing middlewareType=TracingForwarder
time="2020-07-16T19:57:43Z" level=debug msg="Creating middleware" middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix entryPointName=traefik routerName=dashboard@internal
time="2020-07-16T19:57:43Z" level=debug msg="Adding tracing to middleware" middlewareName=dashboard_stripprefix@internal entryPointName=traefik routerName=dashboard@internal
time="2020-07-16T19:57:43Z" level=debug msg="Creating middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex
time="2020-07-16T19:57:43Z" level=debug msg="Setting up redirection from ^(http:\\/\\/[^:\\/]+(:\\d+)?)\\/$ to ${1}/dashboard/" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex
time="2020-07-16T19:57:43Z" level=debug msg="Adding tracing to middleware" routerName=dashboard@internal middlewareName=dashboard_redirect@internal entryPointName=traefik
time="2020-07-16T19:57:43Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=traefik
time="2020-07-16T19:57:43Z" level=debug msg="Creating middleware" middlewareType=Pipelining entryPointName=websecure routerName=production-ingressroutetls-7df7c494de9ca427e2bb@kubernetescrd serviceName=production-ingressroutetls-7df7c494de9ca427e2bb middlewareName=pipelining
time="2020-07-16T19:57:43Z" level=debug msg="Creating load-balancer" routerName=production-ingressroutetls-7df7c494de9ca427e2bb@kubernetescrd serviceName=production-ingressroutetls-7df7c494de9ca427e2bb entryPointName=websecure
time="2020-07-16T19:57:43Z" level=debug msg="Creating server 0 http://10.40.0.2:80" serverName=0 entryPointName=websecure routerName=production-ingressroutetls-7df7c494de9ca427e2bb@kubernetescrd serviceName=production-ingressroutetls-7df7c494de9ca427e2bb
time="2020-07-16T19:57:43Z" level=debug msg="Creating server 1 http://10.44.0.3:80" serverName=1 routerName=production-ingressroutetls-7df7c494de9ca427e2bb@kubernetescrd serviceName=production-ingressroutetls-7df7c494de9ca427e2bb entryPointName=websecure
time="2020-07-16T19:57:43Z" level=debug msg="Added outgoing tracing middleware production-ingressroutetls-7df7c494de9ca427e2bb" routerName=production-ingressroutetls-7df7c494de9ca427e2bb@kubernetescrd middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure
time="2020-07-16T19:57:43Z" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2020-07-16T19:57:43Z" level=debug msg="Adding route for request.whitebox.app with TLS options production-traefik-options" entryPointName=websecure
time="2020-07-16T19:57:44Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-16T19:57:44Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-16T19:57:46Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-16T19:57:46Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-16T19:57:47Z" level=debug msg="http: TLS handshake error from 10.44.0.0:47788: tls: client offered only unsupported versions: []"
time="2020-07-16T19:57:47Z" level=debug msg="http: TLS handshake error from 10.44.0.0:60696: tls: client offered only unsupported versions: []"
time="2020-07-16T19:57:48Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-16T19:57:48Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-16T19:57:49Z" level=debug msg="http: TLS handshake error from 10.44.0.0:60698: tls: client offered only unsupported versions: []"
time="2020-07-16T19:57:50Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd


Hi @jamie,

Could you details how you request traefik ? (client/command line, etc...)

Hi @jbd,

Could I ask what you mean by request Traefik? My setup all runs in Kubernetes via the CLI using yaml files for my deployment. Requests are made via Cloudflare.

https://request.whitebox.app > CloudFlare > Firewall > HAProxy > Traefik

I was asking about the tool used to do the request (browser/curl/etc..)
Could you try without applying the tlsoption ?
Is your HAProxy in TLS Passthrough mode ?

Thanks @jbd for taking the time to take a look at this.

I'm just using my browser to test this. I've removed the tls option and removed it from my ingressroute.

  tls:
    secretName: default-certificate

HAProxy should just be passing TCP sessions through.

global
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    maxconn 4096
    user haproxy
    group haproxy
    daemon


defaults
    log     global
    mode    tcp
    option  tcplog
    option  dontlognull
    timeout connect 180s
    timeout client  180s
    timeout server  180s
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http


frontend localhost80
    bind *:80
    mode http
    redirect scheme https code 301 if !{ ssl_fc }

frontend localhost443
    bind *:443
    option tcplog
    mode tcp

    acl tls req.ssl_hello_type 1

    tcp-request inspect-delay 5s
    tcp-request content accept if tls
    default_backend wordpress_cluster

backend wordpress_cluster
    mode tcp

    option ssl-hello-chk

    server worker-1 10.72.100.1:30443 check
    server worker-2 10.72.100.2:30443 check

Same problem with TLS option removed

time="2020-07-17T10:00:57Z" level=info msg="Configuration loaded from flags."
time="2020-07-17T10:00:57Z" level=info msg="Traefik version 2.2.1 built on 2020-04-29T18:02:09Z"
time="2020-07-17T10:00:57Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"traefik\":{\"address\":\":8080\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{},\"http\":{}},\"web\":{\"address\":\":8000\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{},\"http\":{}},\"websecure\":{\"address\":\":4443\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{},\"http\":{}}},\"providers\":{\"providersThrottleDuration\":2000000000,\"kubernetesCRD\":{}},\"api\":{\"insecure\":true,\"dashboard\":true},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"},\"accessLog\":{\"format\":\"common\",\"filters\":{},\"fields\":{\"defaultMode\":\"keep\",\"headers\":{\"defaultMode\":\"drop\"}}}}"
time="2020-07-17T10:00:57Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/contributing/data-collection/\n"
time="2020-07-17T10:00:57Z" level=debug msg="Start TCP Server" entryPointName=traefik
time="2020-07-17T10:00:57Z" level=info msg="Starting provider aggregator.ProviderAggregator {}"
time="2020-07-17T10:00:57Z" level=debug msg="Start TCP Server" entryPointName=web
time="2020-07-17T10:00:57Z" level=debug msg="Start TCP Server" entryPointName=websecure
time="2020-07-17T10:00:57Z" level=info msg="Starting provider *traefik.Provider {}"
time="2020-07-17T10:00:57Z" level=info msg="Starting provider *crd.Provider {}"
time="2020-07-17T10:00:57Z" level=debug msg="Using label selector: \"\"" providerName=kubernetescrd
time="2020-07-17T10:00:57Z" level=info msg="label selector is: \"\"" providerName=kubernetescrd
time="2020-07-17T10:00:57Z" level=info msg="Creating in-cluster Provider client" providerName=kubernetescrd
time="2020-07-17T10:00:57Z" level=debug msg="Configuration received from provider internal: {\"http\":{\"routers\":{\"api\":{\"entryPoints\":[\"traefik\"],\"service\":\"api@internal\",\"rule\":\"PathPrefix(`/api`)\",\"priority\":2147483646},\"dashboard\":{\"entryPoints\":[\"traefik\"],\"middlewares\":[\"dashboard_redirect@internal\",\"dashboard_stripprefix@internal\"],\"service\":\"dashboard@internal\",\"rule\":\"PathPrefix(`/`)\",\"priority\":2147483645}},\"services\":{\"api\":{},\"dashboard\":{},\"noop\":{}},\"middlewares\":{\"dashboard_redirect\":{\"redirectRegex\":{\"regex\":\"^(http:\\\\/\\\\/[^:\\\\/]+(:\\\\d+)?)\\\\/$\",\"replacement\":\"${1}/dashboard/\",\"permanent\":true}},\"dashboard_stripprefix\":{\"stripPrefix\":{\"prefixes\":[\"/dashboard/\",\"/dashboard\"]}}}},\"tcp\":{},\"tls\":{}}" providerName=internal
time="2020-07-17T10:00:57Z" level=debug msg="Added outgoing tracing middleware api@internal" entryPointName=traefik routerName=api@internal middlewareName=tracing middlewareType=TracingForwarder
time="2020-07-17T10:00:57Z" level=debug msg="Added outgoing tracing middleware dashboard@internal" entryPointName=traefik routerName=dashboard@internal middlewareType=TracingForwarder middlewareName=tracing
time="2020-07-17T10:00:57Z" level=debug msg="Creating middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix
time="2020-07-17T10:00:57Z" level=debug msg="Adding tracing to middleware" routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal entryPointName=traefik
time="2020-07-17T10:00:57Z" level=debug msg="Creating middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex
time="2020-07-17T10:00:57Z" level=debug msg="Setting up redirection from ^(http:\\/\\/[^:\\/]+(:\\d+)?)\\/$ to ${1}/dashboard/" middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex entryPointName=traefik routerName=dashboard@internal
time="2020-07-17T10:00:57Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal
time="2020-07-17T10:00:57Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareType=Recovery middlewareName=traefik-internal-recovery
time="2020-07-17T10:00:57Z" level=debug msg="No default certificate, generating one"
time="2020-07-17T10:00:58Z" level=debug msg="Configuration received from provider kubernetescrd: {\"http\":{},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=kubernetescrd
time="2020-07-17T10:00:58Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:00:58Z" level=debug msg="http: panic serving 10.40.0.0:42092: runtime error: invalid memory address or nil pointer dereference"
time="2020-07-17T10:00:58Z" level=debug msg="goroutine 231 [running]:"
time="2020-07-17T10:00:58Z" level=debug msg="net/http.(*conn).serve.func1(0x419c6c0)"
time="2020-07-17T10:00:58Z" level=debug msg="\t/usr/local/go/src/net/http/server.go:1772 +0xf0"
time="2020-07-17T10:00:58Z" level=debug msg="panic(0x1a96e88, 0x37ccc68)"
time="2020-07-17T10:00:58Z" level=debug msg="\t/usr/local/go/src/runtime/panic.go:975 +0x3d4"
time="2020-07-17T10:00:58Z" level=debug msg="crypto/tls.(*Conn).serverHandshake(0x4234200, 0x0, 0x1bf6458)"
time="2020-07-17T10:00:58Z" level=debug msg="\t/usr/local/go/src/crypto/tls/handshake_server.go:41 +0x38"
time="2020-07-17T10:00:58Z" level=debug msg="crypto/tls.(*Conn).Handshake(0x4234200, 0x0, 0x0)"
time="2020-07-17T10:00:58Z" level=debug msg="\t/usr/local/go/src/crypto/tls/conn.go:1342 +0x200"
time="2020-07-17T10:00:58Z" level=debug msg="net/http.(*conn).serve(0x419c6c0, 0x21d4460, 0x44d37e0)"
time="2020-07-17T10:00:58Z" level=debug msg="\t/usr/local/go/src/net/http/server.go:1788 +0x180"
time="2020-07-17T10:00:58Z" level=debug msg="created by net/http.(*Server).Serve"
time="2020-07-17T10:00:58Z" level=debug msg="\t/usr/local/go/src/net/http/server.go:2933 +0x2d0"
time="2020-07-17T10:00:59Z" level=debug msg="Added outgoing tracing middleware dashboard@internal" routerName=dashboard@internal middlewareName=tracing middlewareType=TracingForwarder entryPointName=traefik
time="2020-07-17T10:00:59Z" level=debug msg="Creating middleware" routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix entryPointName=traefik
time="2020-07-17T10:00:59Z" level=debug msg="Adding tracing to middleware" routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal entryPointName=traefik
time="2020-07-17T10:00:59Z" level=debug msg="Creating middleware" entryPointName=traefik routerName=dashboard@internal middlewareType=RedirectRegex middlewareName=dashboard_redirect@internal
time="2020-07-17T10:00:59Z" level=debug msg="Setting up redirection from ^(http:\\/\\/[^:\\/]+(:\\d+)?)\\/$ to ${1}/dashboard/" entryPointName=traefik routerName=dashboard@internal middlewareType=RedirectRegex middlewareName=dashboard_redirect@internal
time="2020-07-17T10:00:59Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal
time="2020-07-17T10:00:59Z" level=debug msg="Added outgoing tracing middleware api@internal" entryPointName=traefik routerName=api@internal middlewareType=TracingForwarder middlewareName=tracing
time="2020-07-17T10:00:59Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2020-07-17T10:00:59Z" level=debug msg="No default certificate, generating one"
time="2020-07-17T10:00:59Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:00Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:00Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:00Z" level=debug msg="http: TLS handshake error from 10.40.0.0:42098: tls: client offered only unsupported versions: []"
time="2020-07-17T10:01:00Z" level=debug msg="Configuration received from provider kubernetescrd: {\"http\":{\"routers\":{\"production-simpleingressroute-7df7c494de9ca427e2bb\":{\"entryPoints\":[\"web\"],\"service\":\"production-simpleingressroute-7df7c494de9ca427e2bb\",\"rule\":\"Host(`request.whitebox.app`) \\u0026\\u0026 PathPrefix(`/`)\"}},\"services\":{\"production-simpleingressroute-7df7c494de9ca427e2bb\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.40.0.3:80\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=kubernetescrd
time="2020-07-17T10:01:01Z" level=debug msg="Configuration received from provider kubernetescrd: {\"http\":{\"routers\":{\"production-ingressroutetls-7df7c494de9ca427e2bb\":{\"entryPoints\":[\"websecure\"],\"service\":\"production-ingressroutetls-7df7c494de9ca427e2bb\",\"rule\":\"Host(`request.whitebox.app`) \\u0026\\u0026 PathPrefix(`/`)\",\"tls\":{}},\"production-simpleingressroute-7df7c494de9ca427e2bb\":{\"entryPoints\":[\"web\"],\"service\":\"production-simpleingressroute-7df7c494de9ca427e2bb\",\"rule\":\"Host(`request.whitebox.app`) \\u0026\\u0026 PathPrefix(`/`)\"}},\"services\":{\"production-ingressroutetls-7df7c494de9ca427e2bb\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.40.0.3:80\"}],\"passHostHeader\":true}},\"production-simpleingressroute-7df7c494de9ca427e2bb\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.40.0.3:80\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=kubernetescrd
time="2020-07-17T10:01:01Z" level=debug msg="http: TLS handshake error from 10.40.0.0:39282: tls: client offered only unsupported versions: []"
time="2020-07-17T10:01:02Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:02Z" level=debug msg="Configuration received from provider kubernetescrd: {\"http\":{\"routers\":{\"production-ingressroutetls-7df7c494de9ca427e2bb\":{\"entryPoints\":[\"websecure\"],\"service\":\"production-ingressroutetls-7df7c494de9ca427e2bb\",\"rule\":\"Host(`request.whitebox.app`) \\u0026\\u0026 PathPrefix(`/`)\",\"tls\":{}},\"production-simpleingressroute-7df7c494de9ca427e2bb\":{\"entryPoints\":[\"web\"],\"service\":\"production-simpleingressroute-7df7c494de9ca427e2bb\",\"rule\":\"Host(`request.whitebox.app`) \\u0026\\u0026 PathPrefix(`/`)\"}},\"services\":{\"production-ingressroutetls-7df7c494de9ca427e2bb\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.40.0.3:80\"},{\"url\":\"http://10.44.0.3:80\"}],\"passHostHeader\":true}},\"production-simpleingressroute-7df7c494de9ca427e2bb\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.40.0.3:80\"},{\"url\":\"http://10.44.0.3:80\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=kubernetescrd
time="2020-07-17T10:01:02Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:03Z" level=debug msg="Creating middleware" middlewareType=Pipelining routerName=production-simpleingressroute-7df7c494de9ca427e2bb@kubernetescrd serviceName=production-simpleingressroute-7df7c494de9ca427e2bb entryPointName=web middlewareName=pipelining
time="2020-07-17T10:01:03Z" level=debug msg="Creating load-balancer" routerName=production-simpleingressroute-7df7c494de9ca427e2bb@kubernetescrd serviceName=production-simpleingressroute-7df7c494de9ca427e2bb entryPointName=web
time="2020-07-17T10:01:03Z" level=debug msg="Creating server 0 http://10.40.0.3:80" serviceName=production-simpleingressroute-7df7c494de9ca427e2bb serverName=0 entryPointName=web routerName=production-simpleingressroute-7df7c494de9ca427e2bb@kubernetescrd
time="2020-07-17T10:01:03Z" level=debug msg="Added outgoing tracing middleware production-simpleingressroute-7df7c494de9ca427e2bb" routerName=production-simpleingressroute-7df7c494de9ca427e2bb@kubernetescrd middlewareName=tracing middlewareType=TracingForwarder entryPointName=web
time="2020-07-17T10:01:03Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=web middlewareName=traefik-internal-recovery
time="2020-07-17T10:01:03Z" level=debug msg="Added outgoing tracing middleware api@internal" entryPointName=traefik routerName=api@internal middlewareName=tracing middlewareType=TracingForwarder
time="2020-07-17T10:01:03Z" level=debug msg="Added outgoing tracing middleware dashboard@internal" middlewareType=TracingForwarder entryPointName=traefik routerName=dashboard@internal middlewareName=tracing
time="2020-07-17T10:01:03Z" level=debug msg="Creating middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix
time="2020-07-17T10:01:03Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal
time="2020-07-17T10:01:03Z" level=debug msg="Creating middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex
time="2020-07-17T10:01:03Z" level=debug msg="Setting up redirection from ^(http:\\/\\/[^:\\/]+(:\\d+)?)\\/$ to ${1}/dashboard/" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex
time="2020-07-17T10:01:03Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik middlewareName=dashboard_redirect@internal routerName=dashboard@internal
time="2020-07-17T10:01:03Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=traefik
time="2020-07-17T10:01:03Z" level=debug msg="No default certificate, generating one"
time="2020-07-17T10:01:03Z" level=debug msg="http: TLS handshake error from 10.40.0.0:39290: tls: client offered only unsupported versions: []"
time="2020-07-17T10:01:04Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:04Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:05Z" level=debug msg="http: TLS handshake error from 10.40.0.0:39298: tls: client offered only unsupported versions: []"
time="2020-07-17T10:01:06Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:06Z" level=debug msg="http: TLS handshake error from 10.40.0.0:42118: tls: client offered only unsupported versions: []"
time="2020-07-17T10:01:06Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:07Z" level=debug msg="No store is defined to add the certificate MIIEpDCCA4ygAwIBAgIUX8H4DzPqZYSKmY8ILLsf8Qv/lMQwDQ, it will be added to the default store."
time="2020-07-17T10:01:07Z" level=debug msg="Adding certificate for domain(s) cloudflare origin certificate,*.whitebox.app,whitebox.app"
time="2020-07-17T10:01:07Z" level=debug msg="No default certificate, generating one"
time="2020-07-17T10:01:08Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:09Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:10Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:11Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:12Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:12Z" level=debug msg="http: TLS handshake error from 10.40.0.0:42148: tls: client offered only unsupported versions: []"
time="2020-07-17T10:01:13Z" level=debug msg="Added outgoing tracing middleware dashboard@internal" middlewareType=TracingForwarder entryPointName=traefik routerName=dashboard@internal middlewareName=tracing
time="2020-07-17T10:01:13Z" level=debug msg="Creating middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix
time="2020-07-17T10:01:13Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal
time="2020-07-17T10:01:13Z" level=debug msg="Creating middleware" middlewareType=RedirectRegex entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal
time="2020-07-17T10:01:13Z" level=debug msg="Setting up redirection from ^(http:\\/\\/[^:\\/]+(:\\d+)?)\\/$ to ${1}/dashboard/" middlewareType=RedirectRegex entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal
time="2020-07-17T10:01:13Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal
time="2020-07-17T10:01:13Z" level=debug msg="Added outgoing tracing middleware api@internal" middlewareName=tracing middlewareType=TracingForwarder routerName=api@internal entryPointName=traefik
time="2020-07-17T10:01:13Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareType=Recovery middlewareName=traefik-internal-recovery
time="2020-07-17T10:01:13Z" level=debug msg="Creating middleware" middlewareType=Pipelining entryPointName=web routerName=production-simpleingressroute-7df7c494de9ca427e2bb@kubernetescrd serviceName=production-simpleingressroute-7df7c494de9ca427e2bb middlewareName=pipelining
time="2020-07-17T10:01:13Z" level=debug msg="Creating load-balancer" entryPointName=web routerName=production-simpleingressroute-7df7c494de9ca427e2bb@kubernetescrd serviceName=production-simpleingressroute-7df7c494de9ca427e2bb
time="2020-07-17T10:01:13Z" level=debug msg="Creating server 0 http://10.40.0.3:80" entryPointName=web routerName=production-simpleingressroute-7df7c494de9ca427e2bb@kubernetescrd serviceName=production-simpleingressroute-7df7c494de9ca427e2bb serverName=0
time="2020-07-17T10:01:13Z" level=debug msg="Creating server 1 http://10.44.0.3:80" entryPointName=web routerName=production-simpleingressroute-7df7c494de9ca427e2bb@kubernetescrd serviceName=production-simpleingressroute-7df7c494de9ca427e2bb serverName=1
time="2020-07-17T10:01:13Z" level=debug msg="Added outgoing tracing middleware production-simpleingressroute-7df7c494de9ca427e2bb" entryPointName=web routerName=production-simpleingressroute-7df7c494de9ca427e2bb@kubernetescrd middlewareName=tracing middlewareType=TracingForwarder
time="2020-07-17T10:01:13Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2020-07-17T10:01:13Z" level=debug msg="Creating middleware" middlewareName=pipelining middlewareType=Pipelining entryPointName=websecure routerName=production-ingressroutetls-7df7c494de9ca427e2bb@kubernetescrd serviceName=production-ingressroutetls-7df7c494de9ca427e2bb
time="2020-07-17T10:01:13Z" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=production-ingressroutetls-7df7c494de9ca427e2bb@kubernetescrd serviceName=production-ingressroutetls-7df7c494de9ca427e2bb
time="2020-07-17T10:01:13Z" level=debug msg="Creating server 0 http://10.40.0.3:80" entryPointName=websecure routerName=production-ingressroutetls-7df7c494de9ca427e2bb@kubernetescrd serviceName=production-ingressroutetls-7df7c494de9ca427e2bb serverName=0
time="2020-07-17T10:01:13Z" level=debug msg="Creating server 1 http://10.44.0.3:80" routerName=production-ingressroutetls-7df7c494de9ca427e2bb@kubernetescrd serverName=1 serviceName=production-ingressroutetls-7df7c494de9ca427e2bb entryPointName=websecure
time="2020-07-17T10:01:13Z" level=debug msg="Added outgoing tracing middleware production-ingressroutetls-7df7c494de9ca427e2bb" middlewareType=TracingForwarder entryPointName=websecure routerName=production-ingressroutetls-7df7c494de9ca427e2bb@kubernetescrd middlewareName=tracing
time="2020-07-17T10:01:13Z" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2020-07-17T10:01:13Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:14Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:15Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:16Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:17Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:18Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:19Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:21Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:21Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:22Z" level=debug msg="http: TLS handshake error from 10.40.0.0:42190: tls: client offered only unsupported versions: []"
time="2020-07-17T10:01:23Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:23Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:23Z" level=debug msg="http: TLS handshake error from 10.40.0.0:39354: tls: client offered only unsupported versions: []"
time="2020-07-17T10:01:24Z" level=debug msg="http: TLS handshake error from 10.40.0.0:42194: tls: client offered only unsupported versions: []"
time="2020-07-17T10:01:25Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:25Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:25Z" level=debug msg="http: TLS handshake error from 10.40.0.0:39366: tls: client offered only unsupported versions: []"
time="2020-07-17T10:01:26Z" level=debug msg="http: TLS handshake error from 10.40.0.0:42202: tls: client offered only unsupported versions: []"
time="2020-07-17T10:01:27Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:27Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-17T10:01:27Z" level=debug msg="http: TLS handshake error from 10.40.0.0:39372: tls: client offered only unsupported versions: []"
time="2020-07-17T10:01:28Z" level=debug msg="http: TLS handshake error from 10.40.0.0:42212: tls: client offered only unsupported versions: []"

When making requests it would appear it's also causing 504 Gateway Timeouts

time="2020-07-17T10:22:30Z" level=debug msg="'504 Gateway Timeout' caused by: dial tcp 10.44.0.3:80: i/o timeout"
time="2020-07-17T10:22:30Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept-Encoding\":[\"gzip\"],\"Cdn-Loop\":[\"cloudflare\"],\"Cf-Connecting-Ip\":[\"147.75.193.249\"],\"Cf-Ipcountry\":[\"US\"],\"Cf-Ray\":[\"5b433d2eab45f055-EWR\"],\"Cf-Request-Id\":[\"03fde491240000f055429db200000001\"],\"Cf-Visitor\":[\"{\\\"scheme\\\":\\\"https\\\"}\"],\"Connection\":[\"Keep-Alive\"],\"User-Agent\":[\"worldping-api\"],\"X-Forwarded-Host\":[\"request.whitebox.app\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-6d8f775b57-9trnt\"],\"X-Real-Ip\":[\"10.40.0.0\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"request.whitebox.app\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.40.0.0:46544\",\"RequestURI\":\"/\",\"TLS\":null}"
10.40.0.0 - - [17/Jul/2020:10:22:00 +0000] "GET / HTTP/1.1" 504 15 "-" "-" 12 "production-ingressroutetls-7df7c494de9ca427e2bb@kubernetescrd" "http://10.44.0.3:80" 30001ms

I'm investigating,
my first thought was that the client side version of TLS was lower than TLS1.2 or that there was an error in a configuration between the client and Traefik.

1 Like

I'm sorry, I can't reproduce your issue. I need a complete reproducible test case or environment to help you.

At least, check the version of TLS used by your client, it seems to be an unknown/invalid version.
For example if we try with an unsupported version like :
curl -vvv --tls-max 1.0 -k -L https://mydomain

We have a corresponding Traefik log :
level=debug msg="http: TLS handshake error from 10.42.0.1:39063: tls: client offered only unsupported versions: [301]"

So, it looks like that your client does not provide any TLS version.

1 Like

Thanks for taking a look, it's only my test environment so I'm happy to give you access to my environment if you have the time? If so what's the best way to get the details etc over to you?

Have you check the TLS version used ?
Could you provide a full configuration of your environments ?

curl -vvv --tls-max 1.0 -k -L https://request.whitebox.app

Just hangs.

time="2020-07-22T14:26:27Z" level=debug msg="'504 Gateway Timeout' caused by: dial tcp 10.40.0.1:80: i/o timeout"
time="2020-07-22T14:26:27Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept-Encoding\":[\"gzip\"],\"Cdn-Loop\":[\"cloudflare\"],\"Cf-Connecting-Ip\":[\"35.196.53.64\"],\"Cf-Ipcountry\":[\"US\"],\"Cf-Ray\":[\"5b6dd579b90aea4d-IAD\"],\"Cf-Request-Id\":[\"041883c0100000ea4d3b900200000001\"],\"Cf-Visitor\":[\"{\\\"scheme\\\":\\\"https\\\"}\"],\"Connection\":[\"Keep-Alive\"],\"User-Agent\":[\"worldping-api\"],\"X-Forwarded-Host\":[\"request.whitebox.app\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-6d8f775b57-bzmgz\"],\"X-Real-Ip\":[\"10.44.0.0\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"request.whitebox.app\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.44.0.0:57702\",\"RequestURI\":\"/\",\"TLS\":null}"
10.44.0.0 - - [22/Jul/2020:14:25:57 +0000] "GET / HTTP/1.1" 504 15 "-" "-" 1 "production-ingressroutetls-7df7c494de9ca427e2bb@kubernetescrd" "http://10.40.0.1:80" 30003ms
time="2020-07-22T14:26:28Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-22T14:26:28Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd

Environment Flow

  1. User
  2. CloudFlare
  3. Public NAT on Firewall to Keepalived HAProxy IP
  4. HAProxy TCP Pass through to Traefik NodePort.
  5. Traefik TLS Offload using CloudFlare Origin Certificate.

Is there specific things about my environment you need to know?