# How do I diagnose a Traefik2 error when websecure is used and errors are encountered

**URL:** <https://community.traefik.io/t/how-do-i-diagnose-a-traefik2-error-when-websecure-is-used-and-errors-are-encountered/2864>\
**Category:** Traefik v2\
**Tags:** letsencrypt-acme\
**Created:** [November 7, 2019, 9:35pm UTC](https://community.traefik.io/t/how-do-i-diagnose-a-traefik2-error-when-websecure-is-used-and-errors-are-encountered/2864 "2019-11-07T21:35:55Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![mojochao](https://sea2.discourse-cdn.com/flex020/user_avatar/community.traefik.io/mojochao/32/1080_2.png) [@mojochao](https://community.traefik.io/u/mojochao)\
**Post date:** [November 7, 2019, 9:35pm UTC](https://community.traefik.io/t/how-do-i-diagnose-a-traefik2-error-when-websecure-is-used-and-errors-are-encountered/2864/1 "2019-11-07T21:35:55Z")

</div>

I have installed Traefik 2. It's deployment looks like the following:

```auto
λ k describe deployment -n kube-system traefik
Name: traefik
Namespace: kube-system
CreationTimestamp: Tue, 05 Nov 2019 13:20:57 -0600
Labels: app=traefik
                        chart=traefik-2.0.0
                        heritage=Tiller
                        release=traefik
Annotations: deployment.kubernetes.io/revision: 2
Selector: app=traefik,release=traefik
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
  Labels: app=traefik
                    chart=traefik-2.0.0
                    heritage=Tiller
                    release=traefik
  Service Account: traefik
  Containers:
   traefik:
    Image: traefik:2.0.4
    Ports: 8000/TCP, 8443/TCP, 9000/TCP
    Host Ports: 0/TCP, 0/TCP, 0/TCP
    Args:
      --entryPoints.web.address=:8000
      --entryPoints.websecure.address=:8443
      --entryPoints.traefik.address=:9000
      --api.dashboard=true
      --api.insecure=true
      --ping=true
      --providers.kubernetescrd
      --log.level=INFO
      --certificatesresolvers.default.acme.tlschallenge
      --certificatesresolvers.default.acme.email=agooch@samba.tv
      --certificatesresolvers.default.acme.storage=acme.json
      --certificatesresolvers.default.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
    Limits:
      cpu: 300m
      memory: 150Mi
    Requests:
      cpu: 100m
      memory: 50Mi
    Liveness: http-get http://:9000/ping delay=10s timeout=2s period=10s #success=1 #failure=3
    Readiness: http-get http://:9000/ping delay=10s timeout=2s period=10s #success=1 #failure=1
    Environment: <none>
    Mounts: <none>
  Volumes: <none>
Conditions:
  Type Status Reason
  ---- ------ ------
  Available True MinimumReplicasAvailable
  Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: traefik-77b9ddc5f5 (1/1 replicas created)
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal ScalingReplicaSet 31m deployment-controller Scaled up replica set traefik-77b9ddc5f5 to 1
  Normal ScalingReplicaSet 31m deployment-controller Scaled down replica set traefik-6f867df86d to 0

```

It started up with no error.

```auto
λ k logs -n kube-system traefik-77b9ddc5f5-t7wdk
time="2019-11-07T20:57:40Z" level=info msg="Configuration loaded from flags."
time="2019-11-07T20:57:40Z" level=info msg="Traefik version 2.0.4 built on 2019-10-28T20:23:57Z"
time="2019-11-07T20:57:40Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/v2.0/contributing/data-collection/\n"
time="2019-11-07T20:57:41Z" level=info msg="Starting provider aggregator.ProviderAggregator {}"
time="2019-11-07T20:57:41Z" level=info msg="Starting provider *crd.Provider {}"
time="2019-11-07T20:57:41Z" level=info msg="label selector is: \"\"" providerName=kubernetescrd
time="2019-11-07T20:57:41Z" level=info msg="Creating in-cluster Provider client" providerName=kubernetescrd
time="2019-11-07T20:57:41Z" level=info msg="Starting provider *acme.Provider {\"email\":\"agooch@samba.tv\",\"caServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"storage\":\"acme.json\",\"keyType\":\"RSA4096\",\"tlsChallenge\":{},\"ResolverName\":\"default\",\"store\":{},\"ChallengeStore\":{}}"
time="2019-11-07T20:57:41Z" level=info msg="Testing certificate renew..." providerName=default.acme

```

When I add an ingress route using `websecure` entrypoint I see the following errors in the Traefik logs.

```auto
time="2019-11-07T21:01:17Z" level=error msg="Cannot create service: service not found recsapi/recsapi-rest" namespace=recsapi providerName=kubernetescrd ingress=recsapi-rest serviceName=recsapi-rest servicePort=80
time="2019-11-07T21:01:17Z" level=error msg="the service \"recsapi-recsapi-rest-a431b128267aabc954fd@kubernetescrd\" does not exist" routerName=recsapi-recsapi-rest-a431b128267aabc954fd@kubernetescrd entryPointName=websecure
time="2019-11-07T21:01:17Z" level=error msg="Cannot create service: service not found recsapi/recsapi-rest" providerName=kubernetescrd ingress=recsapi-rest namespace=recsapi serviceName=recsapi-rest servicePort=80
time="2019-11-07T21:03:59Z" level=error msg="Cannot create service: subset not found" ingress=recsapi-rest namespace=recsapi serviceName=recsapi-rest servicePort=80 providerName=kubernetescrd
time="2019-11-07T21:03:59Z" level=error msg="the service \"recsapi-recsapi-rest-a431b128267aabc954fd@kubernetescrd\" does not exist" entryPointName=web routerName=recsapi-recsapi-rest-a431b128267aabc954fd@kubernetescrd
time="2019-11-07T21:04:00Z" level=error msg="Cannot create service: subset not found" namespace=recsapi serviceName=recsapi-rest servicePort=80 providerName=kubernetescrd ingress=recsapi-rest
time="2019-11-07T21:04:00Z" level=error msg="Cannot create service: subset not found" ingress=recsapi-rest serviceName=recsapi-rest servicePort=80 namespace=recsapi providerName=kubernetescrd

```

The `recsapi-rest` service does exist in the `recsapi` namespace, and works if the entrypoint is set to `web`. Its IngressRoute looks like the following.

```auto
λ k describe ingressroute recsapi-rest
Name: recsapi-rest
Namespace: recsapi
Labels: helm.sh/chart=recsapi-rest-1.0.0
Annotations: <none>
API Version: traefik.containo.us/v1alpha1
Kind: IngressRoute
Metadata:
  Creation Timestamp: 2019-11-07T21:03:59Z
  Generation: 2
  Resource Version: 353363
  Self Link: /apis/traefik.containo.us/v1alpha1/namespaces/recsapi/ingressroutes/recsapi-rest
  UID: 1eac7098-01a2-11ea-bc62-024775199da6
Spec:
  Entry Points:
    websecure
  Routes:
    Kind: Rule
    Match: PathPrefix(`/`)
    Services:
      Name: recsapi-rest
      Port: 80
Events: <none>

```

Can anyone shed light on the errors I'm seeing? Like I said the service referred to in the error messages does exist in that namespace.

Many thanks in advance!

---

<div class="post-metadata">

**Author:** ![mojochao](https://sea2.discourse-cdn.com/flex020/user_avatar/community.traefik.io/mojochao/32/1080_2.png) [@mojochao](https://community.traefik.io/u/mojochao)\
**Post date:** [November 7, 2019, 11:44pm UTC](https://community.traefik.io/t/how-do-i-diagnose-a-traefik2-error-when-websecure-is-used-and-errors-are-encountered/2864/2 "2019-11-07T23:44:16Z")

</div>

I bumped my log output to `DEBUG` level, and modified my IngressRoute to look like the following:

```auto
λ k describe ingressroute recsapi-rest
Name: recsapi-rest
Namespace: recsapi
Labels: helm.sh/chart=recsapi-rest-1.0.0
Annotations: <none>
API Version: traefik.containo.us/v1alpha1
Kind: IngressRoute
Metadata:
  Creation Timestamp: 2019-11-07T21:03:59Z
  Generation: 5
  Resource Version: 364896
  Self Link: /apis/traefik.containo.us/v1alpha1/namespaces/recsapi/ingressroutes/recsapi-rest
  UID: 1eac7098-01a2-11ea-bc62-024775199da6
Spec:
  Entry Points:
    websecure
  Routes:
    Kind: Rule
    Match: Host('prod.recsapi.mydomain.com') && PathPrefix(`/`)
    Services:
      Name: recsapi-rest
      Port: 80
    Tls:
      Cert Resolver: default
Events: <none>

```

In the logs I now see the following error:

```auto
time="2019-11-07T23:33:57Z" level=debug msg="Configuration received from provider kubernetescrd: {\"http\":{\"routers\":{\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\":{\"entryPoints\":[\"web\"],\"service\":\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\",\"rule\":\"PathPrefix(`/dashboard`) || PathPrefix(`/api`)\"},\"recsapi-recsapi-rest-8ea78446fefd697e6e9e\":{\"entryPoints\":[\"websecure\"],\"service\":\"recsapi-recsapi-rest-8ea78446fefd697e6e9e\",\"rule\":\"Host('prod.api.recommendations.samba.tv') \\u0026\\u0026 PathPrefix(`/`)\"}},\"services\":{\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.31.49.28:9000\"}],\"passHostHeader\":true}},\"recsapi-recsapi-rest-8ea78446fefd697e6e9e\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.31.24.183:80\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"tls\":{}}" providerName=kubernetescrd
time="2019-11-07T23:33:57Z" level=debug msg="Creating middleware" serviceName=recsapi-recsapi-rest-8ea78446fefd697e6e9e entryPointName=websecure middlewareName=pipelining middlewareType=Pipelining routerName=recsapi-recsapi-rest-8ea78446fefd697e6e9e@kubernetescrd
time="2019-11-07T23:33:57Z" level=debug msg="Creating load-balancer" serviceName=recsapi-recsapi-rest-8ea78446fefd697e6e9e entryPointName=websecure routerName=recsapi-recsapi-rest-8ea78446fefd697e6e9e@kubernetescrd
time="2019-11-07T23:33:57Z" level=debug msg="Creating server 0 http://172.31.24.183:80" entryPointName=websecure routerName=recsapi-recsapi-rest-8ea78446fefd697e6e9e@kubernetescrd serviceName=recsapi-recsapi-rest-8ea78446fefd697e6e9e serverName=0
time="2019-11-07T23:33:57Z" level=debug msg="Added outgoing tracing middleware recsapi-recsapi-rest-8ea78446fefd697e6e9e" entryPointName=websecure routerName=recsapi-recsapi-rest-8ea78446fefd697e6e9e@kubernetescrd middlewareName=tracing middlewareType=TracingForwarder
time="2019-11-07T23:33:57Z" level=error msg="error while parsing rule Host('prod.recsapi.mydomain.com') && PathPrefix(`/`): 1:6: illegal rune literal" routerName=recsapi-recsapi-rest-8ea78446fefd697e6e9e@kubernetescrd entryPointName=websecure
time="2019-11-07T23:33:57Z" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2019-11-07T23:33:57Z" level=debug msg="Creating middleware" routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5 entryPointName=web middlewareName=pipelining middlewareType=Pipelining
time="2019-11-07T23:33:57Z" level=debug msg="Creating load-balancer" entryPointName=web routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5
time="2019-11-07T23:33:57Z" level=debug msg="Creating server 0 http://172.31.49.28:9000" serverName=0 entryPointName=web routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5
time="2019-11-07T23:33:57Z" level=debug msg="Added outgoing tracing middleware kube-system-traefik-dashboard-d012b7f875133eeab4e5" entryPointName=web routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd middlewareName=tracing middlewareType=TracingForwarder
time="2019-11-07T23:33:57Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2019-11-07T23:33:57Z" level=debug msg="No default certificate, generating one"
time="2019-11-07T23:33:58Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-07T23:33:58Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd

```

Specifically, the error is:

```auto
time="2019-11-07T23:33:57Z" level=error msg="error while parsing rule Host('prod.recsapi.mydomain.com') && PathPrefix(`/`): 1:6: illegal rune literal" routerName=recsapi-recsapi-rest-8ea78446fefd697e6e9e@kubernetescrd entryPointName=websecure

```

My usage of the Rule `match` property looks like the example in the docs ([https://docs.traefik.io/user-guides/crd-acme/#traefik-routers](https://docs.traefik.io/user-guides/crd-acme/#traefik-routers)) so I'm confused as to why it doesn't parse. Any pointers greatly appreciated!

---

<div class="post-metadata">

**Author:** ![ldez](https://sea2.discourse-cdn.com/flex020/user_avatar/community.traefik.io/ldez/32/5_2.png) [@ldez](https://community.traefik.io/u/ldez)\
**Post date:** [November 7, 2019, 11:47pm UTC](https://community.traefik.io/t/how-do-i-diagnose-a-traefik2-error-when-websecure-is-used-and-errors-are-encountered/2864/3 "2019-11-07T23:47:53Z")

</div>

Hello,

> [@mojochao](#):
>
> Host('[prod.recsapi.mydomain.com](http://prod.recsapi.mydomain.com)')

The rule (`Match`) must use backtick instead of single quote [Routers | Traefik | v2.0](https://docs.traefik.io/v2.0/routing/routers/#rule)

```auto
Host(`prod.recsapi.mydomain.com`) 

```

---

<div class="post-metadata">

**Author:** ![mojochao](https://sea2.discourse-cdn.com/flex020/user_avatar/community.traefik.io/mojochao/32/1080_2.png) [@mojochao](https://community.traefik.io/u/mojochao)\
**Post date:** [November 7, 2019, 11:57pm UTC](https://community.traefik.io/t/how-do-i-diagnose-a-traefik2-error-when-websecure-is-used-and-errors-are-encountered/2864/4 "2019-11-07T23:57:00Z")

</div>

Thanks @Idez, that got me further. From the logs output after making that change:

```auto
time="2019-11-07T23:48:57Z" level=debug msg="Configuration received from provider kubernetescrd: {\"http\":{\"routers\":{\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\":{\"entryPoints\":[\"web\"],\"service\":\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\",\"rule\":\"PathPrefix(`/dashboard`) || PathPrefix(`/api`)\"},\"recsapi-recsapi-rest-98b3c49be6bf59e574ac\":{\"entryPoints\":[\"websecure\"],\"service\":\"recsapi-recsapi-rest-98b3c49be6bf59e574ac\",\"rule\":\"Host(`prod.recsapi.mydomain.com`) \\u0026\\u0026 PathPrefix(`/`)\"}},\"services\":{\"kube-system-traefik-dashboard-d012b7f875133eeab4e5\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.31.49.28:9000\"}],\"passHostHeader\":true}},\"recsapi-recsapi-rest-98b3c49be6bf59e574ac\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.31.24.183:80\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"tls\":{}}" providerName=kubernetescrd
time="2019-11-07T23:48:57Z" level=debug msg="Creating middleware" middlewareType=Pipelining entryPointName=web routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5 middlewareName=pipelining
time="2019-11-07T23:48:57Z" level=debug msg="Creating load-balancer" entryPointName=web routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5
time="2019-11-07T23:48:57Z" level=debug msg="Creating server 0 http://172.31.49.28:9000" routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd serviceName=kube-system-traefik-dashboard-d012b7f875133eeab4e5 entryPointName=web serverName=0
time="2019-11-07T23:48:57Z" level=debug msg="Added outgoing tracing middleware kube-system-traefik-dashboard-d012b7f875133eeab4e5" middlewareType=TracingForwarder routerName=kube-system-traefik-dashboard-d012b7f875133eeab4e5@kubernetescrd entryPointName=web middlewareName=tracing
time="2019-11-07T23:48:57Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=web middlewareName=traefik-internal-recovery
time="2019-11-07T23:48:57Z" level=debug msg="Creating middleware" middlewareType=Pipelining entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd serviceName=recsapi-recsapi-rest-98b3c49be6bf59e574ac middlewareName=pipelining
time="2019-11-07T23:48:57Z" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd serviceName=recsapi-recsapi-rest-98b3c49be6bf59e574ac
time="2019-11-07T23:48:57Z" level=debug msg="Creating server 0 http://172.31.24.183:80" entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd serviceName=recsapi-recsapi-rest-98b3c49be6bf59e574ac serverName=0
time="2019-11-07T23:48:57Z" level=debug msg="Added outgoing tracing middleware recsapi-recsapi-rest-98b3c49be6bf59e574ac" entryPointName=websecure routerName=recsapi-recsapi-rest-98b3c49be6bf59e574ac@kubernetescrd middlewareName=tracing middlewareType=TracingForwarder
time="2019-11-07T23:48:57Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=websecure middlewareName=traefik-internal-recovery
time="2019-11-07T23:48:57Z" level=debug msg="No default certificate, generating one"
time="2019-11-07T23:48:57Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-11-07T23:48:57Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd

```

However when I try to hit that endpoint I see the following in my logs:

```auto
time="2019-11-07T23:51:32Z" level=debug msg="Serving default certificate for request: \"prod.api.recommendations.samba.tv\""
time="2019-11-07T23:51:32Z" level=debug msg="http: TLS handshake error from 172.31.57.196:23116: remote error: tls: unknown certificate authority"

```