TLS secret not found

Traefik Traefik v2
1

I'm in the process of upgrading to Traefik 2.1.3 and I'm running into the "Error configuring TLS: secret X does not exist" problem. Initially I was seeing this with Ingress objects which are currently working with Traefik 1.7. Certificates are provisioned with cert-manager using LE.

I've installed Traefik from the new Helm chart, RBAC configuration is left to the chart's defaults.

Traefik is running with the following options: (note that I have disable the kubernetesingress provider and create an IngressRoute during the process of debugging.)

    - args:
        - '--global.checknewversion=true'
        - '--global.sendanonymoususage=true'
        - '--entryPoints.prometheus.address=:9100'
        - '--entryPoints.traefik.address=:9000'
        - '--entryPoints.web.address=:8000'
        - '--entryPoints.websecure.address=:8443'
        - '--api.dashboard=true'
        - '--ping=true'
        - '--providers.kubernetescrd'
        - '--log.level=DEBUG'
        - '--api.insecure=true'
        - '--accesslog'
        - '--accesslog.format=json'
        - '--metrics.prometheus'
        - '--metrics.prometheus.entrypoint=prometheus'

IngressRoute:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  annotations:
  creationTimestamp: "2020-01-31T03:03:15Z"
  generation: 1
  name: grafana
  namespace: monitoring
  resourceVersion: "2694347"
  selfLink: /apis/traefik.containo.us/v1alpha1/namespaces/monitoring/ingressroutes/grafana
  uid: 6295fb95-1e44-44c3-925f-3a172e5e01c1
spec:
  routes:
  - kind: Rule
    match: Host(`grafana.somedomain`)
    services:
    - name: prometheus-grafana
      port: 80
  tls:
    secretName: grafana-tls

grafana-tls secret, which I fetched using the traefik service account just to prove that RBAC should be configured correctly.

k get secret -n monitoring grafana-tls --as system:serviceaccount:default:traefik2 -o yaml
apiVersion: v1
data:
  ca.crt: ""
  tls.crt: LS0tLS1CR...
  tls.key: LS0tLS1CR...
kind: Secret
metadata:
  annotations:
    cert-manager.io/alt-names: grafana.somedomain
    cert-manager.io/certificate-name: grafana-tls
    cert-manager.io/common-name: grafana.somedomain
    cert-manager.io/ip-sans: ""
    cert-manager.io/issuer-kind: ClusterIssuer
    cert-manager.io/issuer-name: letsencrypt-prod
    cert-manager.io/uri-sans: ""
  creationTimestamp: "2020-01-18T03:48:44Z"
  name: grafana-tls
  namespace: monitoring
  resourceVersion: "1074952"
  selfLink: /api/v1/namespaces/monitoring/secrets/grafana-tls
  uid: 0dbfa586-968c-4ffd-b489-0093ff83fa46
type: kubernetes.io/tls

I've enabled debug level logging but that doesn't seem to shed any additional light on the situation. Any help or advice is greatly appreciated.

2

Additionally, I have exec'd into the traefik container and installed kubectl which can also successfully enumerate and retrieve secrets across namespaces.