How do I diagnose a Traefik2 error when websecure is used and errors are encountered

I have installed Traefik 2. It's deployment looks like the following:

λ k describe deployment -n kube-system traefik
Name:                   traefik
Namespace:              kube-system
CreationTimestamp:      Tue, 05 Nov 2019 13:20:57 -0600
Labels:                 app=traefik
                        chart=traefik-2.0.0
                        heritage=Tiller
                        release=traefik
Annotations:            deployment.kubernetes.io/revision: 2
Selector:               app=traefik,release=traefik
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app=traefik
                    chart=traefik-2.0.0
                    heritage=Tiller
                    release=traefik
  Service Account:  traefik
  Containers:
   traefik:
    Image:       traefik:2.0.4
    Ports:       8000/TCP, 8443/TCP, 9000/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP
    Args:
      --entryPoints.web.address=:8000
      --entryPoints.websecure.address=:8443
      --entryPoints.traefik.address=:9000
      --api.dashboard=true
      --api.insecure=true
      --ping=true
      --providers.kubernetescrd
      --log.level=INFO
      --certificatesresolvers.default.acme.tlschallenge
      --certificatesresolvers.default.acme.email=agooch@samba.tv
      --certificatesresolvers.default.acme.storage=acme.json
      --certificatesresolvers.default.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
    Limits:
      cpu:     300m
      memory:  150Mi
    Requests:
      cpu:        100m
      memory:     50Mi
    Liveness:     http-get http://:9000/ping delay=10s timeout=2s period=10s #success=1 #failure=3
    Readiness:    http-get http://:9000/ping delay=10s timeout=2s period=10s #success=1 #failure=1
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   traefik-77b9ddc5f5 (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  31m   deployment-controller  Scaled up replica set traefik-77b9ddc5f5 to 1
  Normal  ScalingReplicaSet  31m   deployment-controller  Scaled down replica set traefik-6f867df86d to 0

It started up with no error.

λ k logs -n kube-system traefik-77b9ddc5f5-t7wdk
time="2019-11-07T20:57:40Z" level=info msg="Configuration loaded from flags."
time="2019-11-07T20:57:40Z" level=info msg="Traefik version 2.0.4 built on 2019-10-28T20:23:57Z"
time="2019-11-07T20:57:40Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/v2.0/contributing/data-collection/\n"
time="2019-11-07T20:57:41Z" level=info msg="Starting provider aggregator.ProviderAggregator {}"
time="2019-11-07T20:57:41Z" level=info msg="Starting provider *crd.Provider {}"
time="2019-11-07T20:57:41Z" level=info msg="label selector is: \"\"" providerName=kubernetescrd
time="2019-11-07T20:57:41Z" level=info msg="Creating in-cluster Provider client" providerName=kubernetescrd
time="2019-11-07T20:57:41Z" level=info msg="Starting provider *acme.Provider {\"email\":\"agooch@samba.tv\",\"caServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"storage\":\"acme.json\",\"keyType\":\"RSA4096\",\"tlsChallenge\":{},\"ResolverName\":\"default\",\"store\":{},\"ChallengeStore\":{}}"
time="2019-11-07T20:57:41Z" level=info msg="Testing certificate renew..." providerName=default.acme

When I add an ingress route using websecure entrypoint I see the following errors in the Traefik logs.

time="2019-11-07T21:01:17Z" level=error msg="Cannot create service: service not found recsapi/recsapi-rest" namespace=recsapi providerName=kubernetescrd ingress=recsapi-rest serviceName=recsapi-rest servicePort=80
time="2019-11-07T21:01:17Z" level=error msg="the service \"recsapi-recsapi-rest-a431b128267aabc954fd@kubernetescrd\" does not exist" routerName=recsapi-recsapi-rest-a431b128267aabc954fd@kubernetescrd entryPointName=websecure
time="2019-11-07T21:01:17Z" level=error msg="Cannot create service: service not found recsapi/recsapi-rest" providerName=kubernetescrd ingress=recsapi-rest namespace=recsapi serviceName=recsapi-rest servicePort=80
time="2019-11-07T21:03:59Z" level=error msg="Cannot create service: subset not found" ingress=recsapi-rest namespace=recsapi serviceName=recsapi-rest servicePort=80 providerName=kubernetescrd
time="2019-11-07T21:03:59Z" level=error msg="the service \"recsapi-recsapi-rest-a431b128267aabc954fd@kubernetescrd\" does not exist" entryPointName=web routerName=recsapi-recsapi-rest-a431b128267aabc954fd@kubernetescrd
time="2019-11-07T21:04:00Z" level=error msg="Cannot create service: subset not found" namespace=recsapi serviceName=recsapi-rest servicePort=80 providerName=kubernetescrd ingress=recsapi-rest
time="2019-11-07T21:04:00Z" level=error msg="Cannot create service: subset not found" ingress=recsapi-rest serviceName=recsapi-rest servicePort=80 namespace=recsapi providerName=kubernetescrd

The recsapi-rest service does exist in the recsapi namespace, and works if the entrypoint is set to web. Its IngressRoute looks like the following.

λ k describe ingressroute recsapi-rest
Name:         recsapi-rest
Namespace:    recsapi
Labels:       helm.sh/chart=recsapi-rest-1.0.0
Annotations:  <none>
API Version:  traefik.containo.us/v1alpha1
Kind:         IngressRoute
Metadata:
  Creation Timestamp:  2019-11-07T21:03:59Z
  Generation:          2
  Resource Version:    353363
  Self Link:           /apis/traefik.containo.us/v1alpha1/namespaces/recsapi/ingressroutes/recsapi-rest
  UID:                 1eac7098-01a2-11ea-bc62-024775199da6
Spec:
  Entry Points:
    websecure
  Routes:
    Kind:   Rule
    Match:  PathPrefix(`/`)
    Services:
      Name:  recsapi-rest
      Port:  80
Events:      <none>

Can anyone shed light on the errors I'm seeing? Like I said the service referred to in the error messages does exist in that namespace.

Many thanks in advance!

I bumped my log output to DEBUG level, and modified my IngressRoute to look like the following:

λ k describe ingressroute recsapi-rest
Name:         recsapi-rest
Namespace:    recsapi
Labels:       helm.sh/chart=recsapi-rest-1.0.0
Annotations:  <none>
API Version:  traefik.containo.us/v1alpha1
Kind:         IngressRoute
Metadata:
  Creation Timestamp:  2019-11-07T21:03:59Z
  Generation:          5
  Resource Version:    364896
  Self Link:           /apis/traefik.containo.us/v1alpha1/namespaces/recsapi/ingressroutes/recsapi-rest
  UID:                 1eac7098-01a2-11ea-bc62-024775199da6
Spec:
  Entry Points:
    websecure
  Routes:
    Kind:   Rule
    Match:  Host('prod.recsapi.mydomain.com') && PathPrefix(`/`)
    Services:
      Name:  recsapi-rest
      Port:  80
    Tls:
      Cert Resolver:  default
Events:               <none>

In the logs I now see the following error:

time="2019-11-07T23:33:57Z" level=debug msg="Configuration received from provider kubernetescrd: {\"http\":{\"routers\":{\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\":{\"entryPoints\":[\"web\"],\"service\":\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\",\"rule\":\"PathPrefix(`/dashboard`) || PathPrefix(`/api`)\"},\"recsapi-recsapi-rest-8ea78446fefd697e6e9e\":{\"entryPoints\":[\"websecure\"],\"service\":\"recsapi-recsapi-rest-8ea78446fefd697e6e9e\",\"rule\":\"Host('prod.api.recommendations.samba.tv') \\u0026\\u0026 PathPrefix(`/`)\"}},\"services\":{\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.31.49.28:9000\"}],\"passHostHeader\":true}},\"recsapi-recsapi-rest-8ea78446fefd697e6e9e\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.31.24.183:80\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"tls\":{}}" providerName=kubernetescrd
time="2019-11-07T23:33:57Z" level=debug msg="Creating middleware" serviceName=recsapi-recsapi-rest-8ea78446fefd697e6e9e entryPointName=websecure middlewareName=pipelining middlewareType=Pipelining routerName=recsapi-recsapi-rest-8ea78446fefd697e6e9e@kubernetescrd
time="2019-11-07T23:33:57Z" level=debug msg="Creating load-balancer" serviceName=recsapi-recsapi-rest-8ea78446fefd697e6e9e entryPointName=websecure routerName=recsapi-recsapi-rest-8ea78446fefd697e6e9e@kubernetescrd
time="2019-11-07T23:33:57Z" level=debug msg="Creating server 0 http://172.31.24.183:80" entryPointName=websecure routerName=recsapi-recsapi-rest-8ea78446fefd697e6e9e@kubernetescrd serviceName=recsapi-recsapi-rest-8ea78446fefd697e6e9e serverName=0
time="2019-11-07T23:33:57Z" level=debug msg="Added outgoing tracing middleware recsapi-recsapi-rest-8ea78446fefd697e6e9e" entryPointName=websecure routerName=recsapi-recsapi-rest-8ea78446fefd697e6e9e@kubernetescrd middlewareName=tracing middlewareType=TracingForwarder
time="2019-11-07T23:33:57Z" level=error msg="error while parsing rule Host('prod.recsapi.mydomain.com') && PathPrefix(`/`): 1:6: illegal rune literal" routerName=recsapi-recsapi-rest-8ea78446fefd697e6e9e@kubernetescrd entryPointName=websecure
time="2019-11-07T23:33:57Z" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2019-11-07T23:33:57Z" level=debug msg="Creating middleware" routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5 entryPointName=web middlewareName=pipelining middlewareType=Pipelining
time="2019-11-07T23:33:57Z" level=debug msg="Creating load-balancer" entryPointName=web routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5
time="2019-11-07T23:33:57Z" level=debug msg="Creating server 0 http://172.31.49.28:9000" serverName=0 entryPointName=web routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5
time="2019-11-07T23:33:57Z" level=debug msg="Added outgoing tracing middleware kube-system-traefik-dashboard-d012b7f875133eeab4e5" entryPointName=web routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd middlewareName=tracing middlewareType=TracingForwarder
time="2019-11-07T23:33:57Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2019-11-07T23:33:57Z" level=debug msg="No default certificate, generating one"
time="2019-11-07T23:33:58Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-07T23:33:58Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd

Specifically, the error is:

time="2019-11-07T23:33:57Z" level=error msg="error while parsing rule Host('prod.recsapi.mydomain.com') && PathPrefix(`/`): 1:6: illegal rune literal" routerName=recsapi-recsapi-rest-8ea78446fefd697e6e9e@kubernetescrd entryPointName=websecure

My usage of the Rule match property looks like the example in the docs (https://docs.traefik.io/user-guides/crd-acme/#traefik-routers) so I'm confused as to why it doesn't parse. Any pointers greatly appreciated!

Hello,

The rule (Match) must use backtick instead of single quote https://docs.traefik.io/v2.0/routing/routers/#rule

Host(`prod.recsapi.mydomain.com`) 

Thanks @Idez, that got me further. From the logs output after making that change:

time="2019-11-07T23:48:57Z" level=debug msg="Configuration received from provider kubernetescrd: {\"http\":{\"routers\":{\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\":{\"entryPoints\":[\"web\"],\"service\":\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\",\"rule\":\"PathPrefix(`/dashboard`) || PathPrefix(`/api`)\"},\"recsapi-recsapi-rest-98b3c49be6bf59e574ac\":{\"entryPoints\":[\"websecure\"],\"service\":\"recsapi-recsapi-rest-98b3c49be6bf59e574ac\",\"rule\":\"Host(`prod.recsapi.mydomain.com`) \\u0026\\u0026 PathPrefix(`/`)\"}},\"services\":{\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.31.49.28:9000\"}],\"passHostHeader\":true}},\"recsapi-recsapi-rest-98b3c49be6bf59e574ac\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.31.24.183:80\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"tls\":{}}" providerName=kubernetescrd
time="2019-11-07T23:48:57Z" level=debug msg="Creating middleware" middlewareType=Pipelining entryPointName=web routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5 middlewareName=pipelining
time="2019-11-07T23:48:57Z" level=debug msg="Creating load-balancer" entryPointName=web routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5
time="2019-11-07T23:48:57Z" level=debug msg="Creating server 0 http://172.31.49.28:9000" routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5 entryPointName=web serverName=0
time="2019-11-07T23:48:57Z" level=debug msg="Added outgoing tracing middleware kube-system-traefik-dashboard-d012b7f875133eeab4e5" middlewareType=TracingForwarder routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd entryPointName=web middlewareName=tracing
time="2019-11-07T23:48:57Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=web middlewareName=traefik-internal-recovery
time="2019-11-07T23:48:57Z" level=debug msg="Creating middleware" middlewareType=Pipelining entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd serviceName=recsapi-recsapi-rest-98b3c49be6bf59e574ac middlewareName=pipelining
time="2019-11-07T23:48:57Z" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd serviceName=recsapi-recsapi-rest-98b3c49be6bf59e574ac
time="2019-11-07T23:48:57Z" level=debug msg="Creating server 0 http://172.31.24.183:80" entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd serviceName=recsapi-recsapi-rest-98b3c49be6bf59e574ac serverName=0
time="2019-11-07T23:48:57Z" level=debug msg="Added outgoing tracing middleware recsapi-recsapi-rest-98b3c49be6bf59e574ac" entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd middlewareName=tracing middlewareType=TracingForwarder
time="2019-11-07T23:48:57Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=websecure middlewareName=traefik-internal-recovery
time="2019-11-07T23:48:57Z" level=debug msg="No default certificate, generating one"
time="2019-11-07T23:48:57Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-07T23:48:57Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd

However when I try to hit that endpoint I see the following in my logs:

time="2019-11-07T23:51:32Z" level=debug msg="Serving default certificate for request: \"prod.api.recommendations.samba.tv\""
time="2019-11-07T23:51:32Z" level=debug msg="http: TLS handshake error from 172.31.57.196:23116: remote error: tls: unknown certificate authority"