## The setup
- K3s cluster with mostly default configuration. It comes with T…raefik 2 as the ingress controller.
- cert-manager installed via Helm.
- The following `ClusterIssuer`s:
```yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
namespace: cert-manager
spec:
acme:
email: ...
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt-staging-key
solvers:
- http01:
ingress:
serviceType: ClusterIP
class: traefik-cert-manager
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
namespace: cert-manager
spec:
acme:
email: ...
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt-prod-key
solvers:
- http01:
ingress:
serviceType: ClusterIP
class: traefik-cert-manager
```
- A simple app manifest for app `foo`, like this:
```yaml
apiVersion: v1
kind: Namespace
metadata:
name: foo
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: foo
namespace: foo
labels:
app: foo
spec:
replicas: 1
selector:
matchLabels:
app: foo
template:
metadata:
labels:
app: foo
spec:
volumes:
- name: private-key
secret:
secretName: foo-private-key
defaultMode: 0400
- name: build
emptyDir: { }
containers:
- name: foo
image: foo-image
volumeMounts:
- mountPath: /root/.ssh/id_rsa
name: private-key
readOnly: true
subPath: id_rsa
- mountPath: /build/output
name: build
readOnly: false
- name: nginx
image: nginx:1-alpine
volumeMounts:
- mountPath: /usr/share/nginx/html
name: build
readOnly: true
---
apiVersion: v1
kind: Service
metadata:
name: foo-service
namespace: foo
spec:
type: ClusterIP
ports:
- port: 80
selector:
app: foo
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: foo-ingress
namespace: foo
annotations:
cert-manager.io/cluster-issuer: "letsencrypt-staging"
spec:
tls:
- hosts:
- foo.domain.com
secretName: foo-cert
rules:
- host: foo.domain.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: foo-service
port:
number: 80
---
apiVersion: v1
kind: Secret
data:
id_rsa: LS0t...
metadata:
name: foo-private-key
namespace: foo
```
## What works
- Certificate, certificaterequest, and challenge are created.
- `cm-acme-http-solver-...` ingress is created.
- `cm-acme-http-solver-...` service is created.
- `cm-acme-http-solver-...` pod is created.
## The problem
GET requests for `http://foo.domain.com/.well-known/acme-challenge/...` are sent via the `foo-service` service to the `nginx` container, rather than to the `cm-acme-http-solver` pod. This obviously results in a 404.
I modified the `traefik` deployment to add the `--log.level=DEBUG` arg. The only useful bits I got from there is:
```
time="2021-10-27T22:59:24Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/.well-known/acme-challenge/...\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept-Encoding\":[\"gzip\"],\"Connection\":[\"close\"],\"User-Agent\":[\"cert-manager/v1.5.4 (clean)\"],\"X-Forwarded-Host\":[\"foo.domain.com\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"traefik-846c44fdbd-jbtnd\"],\"X-Real-Ip\":[\"10.42.0.1\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"foo.domain.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.42.0.1:64074\",\"RequestURI\":\"/.well-known/acme-challenge/...\",\"TLS\":null}"
time="2021-10-27T22:59:24Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/.well-known/acme-challenge/...\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept-Encoding\":[\"gzip\"],\"Connection\":[\"close\"],\"User-Agent\":[\"cert-manager/v1.5.4 (clean)\"],\"X-Forwarded-Host\":[\"foo.domain.com\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"traefik-846c44fdbd-jbtnd\"],\"X-Real-Ip\":[\"10.42.0.1\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"foo.domain.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.42.0.1:64074\",\"RequestURI\":\"/.well-known/acme-challenge/...\",\"TLS\":null}" ForwardURL="http://10.42.0.40:80"
time="2021-10-27T22:59:24Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/.well-known/acme-challenge/...\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept-Encoding\":[\"gzip\"],\"Connection\":[\"close\"],\"User-Agent\":[\"cert-manager/v1.5.4 (clean)\"],\"X-Forwarded-Host\":[\"foo.domain.com\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"traefik-846c44fdbd-jbtnd\"],\"X-Real-Ip\":[\"10.42.0.1\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"foo.domain.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.42.0.1:64074\",\"RequestURI\":\"/.well-known/acme-challenge/...\",\"TLS\":null}"
```
`10.42.0.40` is the IP of the pod containing my nginx container, so this is definitely wrong.
## Questions
- Who is to blame here, and how can I find out?
- What determines the order in which Ingresses are matched? Is there some kind of weight? I can't find anything on this in the official k8s documentation.
- Is cert-manager compatible with Traefik 2? Is it supposed to be agnostic with regards to the Ingress Controller?
