LE quietly failing on GKE

Traefik Traefik v2
,
1

I have been trying to figure out what's going on with my setup all day. No luck so far. I started with the traefik-helm-chart and worked forward from there.

Here is my deployment...

---
# Source: traefik/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: traefik
  labels:
    app: traefik
    chart: traefik-3.3.1
    release: "traefik"
    heritage: "Helm"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik
      release: traefik
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  template:
    metadata:
      labels:
        app: traefik
        chart: traefik-3.3.1
        release: "traefik"
        heritage: "Helm"
    spec:
      serviceAccountName: traefik
      terminationGracePeriodSeconds: 60
      hostNetwork: true
      nodeSelector:
        cloud.google.com/gke-nodepool: "ingress"
      tolerations:
        - key: "ingress"
          operator: "Exists"
          effect: "NoExecute"
      containers:
      - image: traefik:2.1.3
        name: traefik
        volumeMounts:
        - name: acme-json
          mountPath: /cert
          subPath: acme.json
        resources:
          limits:
            cpu: 200m
            memory: 30Mi
          requests:
            cpu: 100m
            memory: 20Mi
        readinessProbe:
          httpGet:
            path: /ping
            port: 9000
          failureThreshold: 1
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        livenessProbe:
          httpGet:
            path: /ping
            port: 9000
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        ports:
        - name: "traefik"
          containerPort: 9000
          protocol: TCP
        - name: "web"
          containerPort: 80
          hostPort: 80
          protocol: TCP
        - name: "websecure"
          containerPort: 443
          hostPort: 443
          protocol: TCP
        args:
          - "--global.checknewversion=true"
          - "--global.sendanonymoususage=true"
          - "--entryPoints.traefik.address=:9000"
          - "--entryPoints.web.address=:80"
          - "--entryPoints.websecure.address=:443"
          - "--api.dashboard=true"
          - "--ping=true"
          - "--providers.kubernetescrd=true"
          - "--log.level=DEBUG"
          - "--certificatesresolvers.default.acme.tlschallenge"
          - "--certificatesresolvers.default.acme.email=user@domain.com"
          - "--certificatesresolvers.default.acme.storage=/cert/acme.json"
          - "--certificatesresolvers.default.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
      volumes:
        - name: acme-json
          persistentVolumeClaim:
            claimName: traefik-acme-pvc

I also deployed whoami with the manifest from the CRD & Let's Encrypt user guide.

And my test ingress route...with domain replaced with domain.com.

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: tls
spec:
  entryPoints:
  - websecure
  routes:
  - kind: Rule
    match: Host(`test.tls.domain.com`) && PathPrefix(`/tls`)
    services:
    - name: whoami
      port: 80
  tls:
    certresolver: default
    domains:
      - main: "test.tls.domain.com"
    options: {}

Logs don't show anything related to the ACME Challenge. But the provider is started...

time="2020-02-05T23:30:28Z" level=info msg="Starting provider *acme.Provider {\"email\":\"user@domain.com\",\"caServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"storage\":\"/cert/acme.json\",\"keyType\":\"RSA4096\",\"tlsChallenge\":{},\"ResolverName\":\"default\",\"store\":{},\"ChallengeStore\":{}}"

And also, whatever this means:

time="2020-02-05T23:30:28Z" level=info msg="Testing certificate renew..." providerName=default.acme

Then I get the same couple debug lines 1-3 times per second.

time="2020-02-05T23:57:51Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-02-05T23:57:51Z" level=debug msg="No secret name provided" providerName=kubernetescrd

The route works when I curl -k https://test.tls.domain.com. However, it uses the Traefik Default Cert, and not the LE Staging certificate.

I do have another ingressroute working with a certificate saved in a secret, but I am hoping to automate with LE.

I have tried every variation I can think of for the tls section. Starting with just the certresolver: default and working through additional options based on other posts I've found here. Nothing seems to trigger the challenge.

Any help is much, much appreciated.

EDIT: Staging certificate issued correctly if I put the whoami service, deployment, and ingressroute in a different namespace. Is there something at the namespace level that forces it to skip LE if there's also a secret set up?