I'm trying to setup traefik on digital ocean kubernetes, to front a backend service and auto-generate the SSL certificate with let's encrypt using http-challenge. Using DNS challenge is not an option as multiple domains that I don't own will point to the traefik LB.
Sadly, the certificate is always broken, no matter what tweaks I do
I run on traefik:v2.0
What is your environment & configuration (arguments, toml, provider, platform, ...)?
This is the Traefik pre-requisite configs I've setup in Kubernetes
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: ingressroutes.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: IngressRoute
plural: ingressroutes
singular: ingressroute
scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: ingressroutetcps.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: IngressRouteTCP
plural: ingressroutetcps
singular: ingressroutetcp
scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: middlewares.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: Middleware
plural: middlewares
singular: middleware
scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: tlsoptions.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: TLSOption
plural: tlsoptions
singular: tlsoption
scope: Namespaced
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-ingress-controller
rules:
- apiGroups:
- ""
resources:
- services
- endpoints
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- extensions
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- extensions
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- traefik.containo.us
resources:
- middlewares
verbs:
- get
- list
- watch
- apiGroups:
- traefik.containo.us
resources:
- ingressroutes
verbs:
- get
- list
- watch
- apiGroups:
- traefik.containo.us
resources:
- ingressroutetcps
verbs:
- get
- list
- watch
- apiGroups:
- traefik.containo.us
resources:
- tlsoptions
verbs:
- get
- list
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-ingress-controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
name: traefik-ingress-controller
namespace: default
---
apiVersion: v1
kind: ServiceAccount
metadata:
namespace: default
name: traefik-ingress-controller
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: traefik-data
namespace: default
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
The actual traefik deployment file looks like this
# traefik exposure on node port
---
apiVersion: v1
kind: Service
metadata:
name: traefik
spec:
type: NodePort
ports:
- protocol: TCP
name: web
port: 80
nodePort: 30080
- protocol: TCP
name: admin
port: 8080
nodePort: 30081
- protocol: TCP
name: websecure
port: 443
nodePort: 30443
selector:
app: traefik
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
namespace: default
name: traefik
labels:
app: traefik
spec:
replicas: 1
selector:
matchLabels:
app: traefik
template:
metadata:
labels:
app: traefik
spec:
serviceAccountName: traefik-ingress-controller
containers:
- name: traefik
image: traefik:v2.0
args:
- --global.sendAnonymousUsage=false
- --log.level=DEBUG
- --api
- --accesslog
- --entryPoints.web.address=:80
- --entryPoints.web.forwardedheaders.insecure=true
- --entryPoints.websecure.address=:443
- --providers.kubernetescrd
- --ping
- --accesslog=true
- --log=true
- --certificatesResolvers.default.acme.email=myemail@gmail.com
- --certificatesResolvers.default.acme.storage=/var/lib/traefik/acme.json
- --certificatesResolvers.default.acme.caServer=https://acme-v02.api.letsencrypt.org/directory
- --certificatesResolvers.default.acme.keyType=RSA4096
- --certificatesResolvers.default.acme.httpChallenge.entryPoint=web
ports:
- name: web
containerPort: 80
- name: websecure
containerPort: 443
- name: admin
containerPort: 8080
volumeMounts:
- name: traefik-data
mountPath: /var/lib/traefik
volumes:
- name: traefik-data
persistentVolumeClaim:
claimName: traefik-data
IngressRoutes to configure Traefik load balancing routing
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: web
spec:
entryPoints:
- web
routes:
- match: Host(`do.msj.world`) && PathPrefix(`/`)
kind: Rule
priority: 1
services:
- name: whoami-service
port: 9001
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: websecure
spec:
entryPoints:
- websecure
tls:
certResolver: default
routes:
- match: Host(`do.msj.world`) && PathPrefix(`/`)
kind: Rule
priority: 1
services:
- name: whoami-service
port: 9001
the whoami.yml configs
apiVersion: v1
kind: Service
metadata:
name: whoami-service #nom du service
spec:
type: ClusterIP
selector:
app: whoami-deployment
ports:
- name: whoami-port
protocol: TCP
port: 9001
targetPort: 80 # PORT DU DEPLOYMENT
- name: whoami-port-ssl
protocol: TCP
port: 9002
targetPort: 443 # PORT DU DEPLOYMENT
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: whoami-deployment
spec:
selector:
matchLabels:
app: whoami-deployment # répond a ce nom la
replicas: 1
template:
metadata:
labels:
app: whoami-deployment
spec:
imagePullSecrets:
- name: regcred
containers:
- name: whoami-containers
image: containous/whoami # path vers l'image dans repository
ports:
- containerPort: 80
- containerPort: 443
LB and firewall are pretty straight forward, here's the terraform file
resource "digitalocean_loadbalancer" "kubernetes-cl01-public" {
name = "kubernetes-cl01-public"
region = "nyc1"
enable_proxy_protocol = false
#redirect_http_to_https = true
forwarding_rule {
entry_port = 80
entry_protocol = "http"
target_port = 30080
target_protocol = "http"
}
forwarding_rule {
entry_port = 81
entry_protocol = "http"
target_port = 30081
target_protocol = "http"
}
forwarding_rule {
entry_port = 443
entry_protocol = "tcp"
target_port = 30443
target_protocol = "tcp"
#tls_passthrough = true
}
healthcheck {
port = 30081
protocol = "http"
path = "/ping"
}
droplet_tag = "eks-prod-nodes"
}
resource "digitalocean_firewall" "kubernetes-cl01-public" {
name = "kubernetes-cl01-public"
tags = ["eks-prod-nodes"]
inbound_rule {
protocol = "tcp"
port_range = "30080"
source_load_balancer_uids = ["${digitalocean_loadbalancer.kubernetes-cl01-public.id}"]
}
inbound_rule {
protocol = "tcp"
port_range = "30443"
source_load_balancer_uids = ["${digitalocean_loadbalancer.kubernetes-cl01-public.id}"]
}
# admin port
inbound_rule {
protocol = "tcp"
port_range = "30081"
source_load_balancer_uids = ["${digitalocean_loadbalancer.kubernetes-cl01-public.id}"]
source_addresses = ["184.162.220.230"]
}
}
admin panel is working (I'll secure it later by not exposing the port): http://159.203.156.113:81/dashboard/#/
https access with broken certificate: https://do.msj.world/
If applicable, please paste the log output in DEBUG level ( --log.level=DEBUG switch)
time="2019-09-08T20:45:05Z" level=info msg="Configuration loaded from flags."
time="2019-09-08T20:45:05Z" level=info msg="Traefik version 2.0.0-rc2 built on 2019-09-03T20:10:11Z"
time="2019-09-08T20:45:05Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true,\"sendAnonymousUsage\":false},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"traefik\":{\"address\":\":8080\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{}},\"web\":{\"address\":\":80\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{\"insecure\":true}},\"websecure\":{\"address\":\":443\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{}}},\"providers\":{\"providersThrottleDuration\":2000000000,\"kubernetesCRD\":{}},\"api\":{\"dashboard\":true},\"ping\":{},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"},\"accessLog\":{\"format\":\"common\",\"filters\":{},\"fields\":{\"defaultMode\":\"keep\",\"headers\":{\"defaultMode\":\"drop\"}}},\"certificatesResolvers\":{\"default\":{\"acme\":{\"email\":\"cyrjeanmichael@gmail.com\",\"caServer\":\"https://acme-v02.api.letsencrypt.org/directory\",\"storage\":\"/var/lib/traefik/acme.json\",\"keyType\":\"RSA4096\",\"httpChallenge\":{\"entryPoint\":\"web\"}}}}}"
time="2019-09-08T20:45:05Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/v2.0/contributing/data-collection/\n"
time="2019-09-08T20:45:05Z" level=debug msg="No default certificate, generating one"
time="2019-09-08T20:45:05Z" level=info msg="Starting provider aggregator.ProviderAggregator {}"
time="2019-09-08T20:45:05Z" level=debug msg="Start TCP Server" entryPointName=web
time="2019-09-08T20:45:05Z" level=debug msg="Start TCP Server" entryPointName=traefik
time="2019-09-08T20:45:05Z" level=debug msg="Start TCP Server" entryPointName=websecure
time="2019-09-08T20:45:05Z" level=info msg="Starting provider *acme.Provider {\"email\":\"cyrjeanmichael@gmail.com\",\"caServer\":\"https://acme-v02.api.letsencrypt.org/directory\",\"storage\":\"/var/lib/traefik/acme.json\",\"keyType\":\"RSA4096\",\"httpChallenge\":{\"entryPoint\":\"web\"},\"ResolverName\":\"default\",\"store\":{},\"ChallengeStore\":{}}"
time="2019-09-08T20:45:05Z" level=info msg="Testing certificate renew..." providerName=default.acme
time="2019-09-08T20:45:05Z" level=info msg="Starting provider *crd.Provider {}"
time="2019-09-08T20:45:05Z" level=debug msg="Using label selector: \"\"" providerName=kubernetescrd
time="2019-09-08T20:45:05Z" level=info msg="label selector is: \"\"" providerName=kubernetescrd
time="2019-09-08T20:45:05Z" level=info msg="Creating in-cluster Provider client" providerName=kubernetescrd
time="2019-09-08T20:45:05Z" level=debug msg="Configuration received from provider default.acme: {\"http\":{},\"tls\":{}}" providerName=default.acme
time="2019-09-08T20:45:05Z" level=debug msg="No default certificate, generating one"
time="2019-09-08T20:45:06Z" level=debug msg="No secret name provided" providerName=kubernetescrd
time="2019-09-08T20:45:06Z" level=debug msg="Configuration received from provider kubernetescrd: {\"http\":{\"routers\":{\"default/web-f35339e15c89b25de38f\":{\"entryPoints\":[\"web\"],\"service\":\"default/web-f35339e15c89b25de38f\",\"rule\":\"Host(`do.msj.world`) \\u0026\\u0026 PathPrefix(`/`)\",\"priority\":1},\"default/websecure-f35339e15c89b25de38f\":{\"entryPoints\":[\"websecure\"],\"service\":\"default/websecure-f35339e15c89b25de38f\",\"rule\":\"Host(`do.msj.world`) \\u0026\\u0026 PathPrefix(`/`)\",\"priority\":1,\"tls\":{\"certResolver\":\"default\"}}},\"services\":{\"default/web-f35339e15c89b25de38f\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.244.0.139:80\"}],\"passHostHeader\":true}},\"default/websecure-f35339e15c89b25de38f\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.244.0.139:80\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"tls\":{}}" providerName=kubernetescrd
time="2019-09-08T20:45:06Z" level=debug msg="Creating middleware" entryPointName=web routerName=default/web-f35339e15c89b25de38f@kubernetescrd serviceName=default/web-f35339e15c89b25de38f middlewareType=Pipelining middlewareName=pipelining
time="2019-09-08T20:45:06Z" level=debug msg="Creating load-balancer" entryPointName=web routerName=default/web-f35339e15c89b25de38f@kubernetescrd serviceName=default/web-f35339e15c89b25de38f
time="2019-09-08T20:45:06Z" level=debug msg="Creating server 0 http://10.244.0.139:80" entryPointName=web routerName=default/web-f35339e15c89b25de38f@kubernetescrd serviceName=default/web-f35339e15c89b25de38f serverName=0
time="2019-09-08T20:45:06Z" level=debug msg="Added outgoing tracing middleware default/web-f35339e15c89b25de38f" middlewareType=TracingForwarder entryPointName=web routerName=default/web-f35339e15c89b25de38f@kubernetescrd middlewareName=tracing
time="2019-09-08T20:45:06Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2019-09-08T20:45:06Z" level=debug msg="Creating middleware" entryPointName=websecure routerName=default/websecure-f35339e15c89b25de38f@kubernetescrd middlewareType=Pipelining middlewareName=pipelining serviceName=default/websecure-f35339e15c89b25de38f
time="2019-09-08T20:45:06Z" level=debug msg="Creating load-balancer" routerName=default/websecure-f35339e15c89b25de38f@kubernetescrd serviceName=default/websecure-f35339e15c89b25de38f entryPointName=websecure
time="2019-09-08T20:45:06Z" level=debug msg="Creating server 0 http://10.244.0.139:80" serviceName=default/websecure-f35339e15c89b25de38f entryPointName=websecure serverName=0 routerName=default/websecure-f35339e15c89b25de38f@kubernetescrd
time="2019-09-08T20:45:06Z" level=debug msg="Added outgoing tracing middleware default/websecure-f35339e15c89b25de38f" entryPointName=websecure routerName=default/websecure-f35339e15c89b25de38f@kubernetescrd middlewareName=tracing middlewareType=TracingForwarder
time="2019-09-08T20:45:06Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=websecure
time="2019-09-08T20:45:06Z" level=debug msg="No default certificate, generating one"
time="2019-09-08T20:45:06Z" level=debug msg="No secret name provided" providerName=kubernetescrd
time="2019-09-08T20:45:06Z" level=debug msg="Skipping Kubernetes event kind *v1.Secret" providerName=kubernetescrd
time="2019-09-08T20:45:06Z" level=debug msg="No secret name provided" providerName=kubernetescrd
time="2019-09-08T20:45:06Z" level=debug msg="Skipping Kubernetes event kind *v1.Secret" providerName=kubernetescrd
time="2019-09-08T20:45:06Z" level=debug msg="Try to challenge certificate for domain [do.msj.world] founded in HostSNI rule" providerName=default.acme routerName=default/websecure-f35339e15c89b25de38f rule="Host(`do.msj.world`) && PathPrefix(`/`)"
time="2019-09-08T20:45:06Z" level=debug msg="Looking for provided certificate(s) to validate [\"do.msj.world\"]..." rule="Host(`do.msj.world`) && PathPrefix(`/`)" providerName=default.acme routerName=default/websecure-f35339e15c89b25de38f
time="2019-09-08T20:45:06Z" level=debug msg="Domains [\"do.msj.world\"] need ACME certificates generation for domains \"do.msj.world\"." rule="Host(`do.msj.world`) && PathPrefix(`/`)" providerName=default.acme routerName=default/websecure-f35339e15c89b25de38f
time="2019-09-08T20:45:06Z" level=debug msg="Loading ACME certificates [do.msj.world]..." rule="Host(`do.msj.world`) && PathPrefix(`/`)" providerName=default.acme routerName=default/websecure-f35339e15c89b25de38f
time="2019-09-08T20:45:06Z" level=debug msg="No secret name provided" providerName=kubernetescrd
time="2019-09-08T20:45:06Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-09-08T20:45:08Z" level=debug msg="No secret name provided" providerName=kubernetescrd
time="2019-09-08T20:45:08Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-09-08T20:45:10Z" level=debug msg="No secret name provided" providerName=kubernetescrd
time="2019-09-08T20:45:10Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-09-08T20:45:11Z" level=debug msg="Building ACME client..." providerName=default.acme
time="2019-09-08T20:45:11Z" level=debug msg="https://acme-v02.api.letsencrypt.org/directory" providerName=default.acme
time="2019-09-08T20:45:11Z" level=info msg=Register... providerName=default.acme
time="2019-09-08T20:45:11Z" level=debug msg="legolog: [INFO] acme: Registering account for cyrjeanmichael@gmail.com"
time="2019-09-08T20:45:12Z" level=debug msg="Using HTTP Challenge provider." providerName=default.acme
time="2019-09-08T20:45:12Z" level=debug msg="legolog: [INFO] [do.msj.world] acme: Obtaining bundled SAN certificate"
time="2019-09-08T20:45:12Z" level=debug msg="No secret name provided" providerName=kubernetescrd
time="2019-09-08T20:45:12Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-09-08T20:45:13Z" level=debug msg="legolog: [INFO] [do.msj.world] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/253187809"
time="2019-09-08T20:45:13Z" level=debug msg="legolog: [INFO] [do.msj.world] acme: Could not find solver for: tls-alpn-01"
time="2019-09-08T20:45:13Z" level=debug msg="legolog: [INFO] [do.msj.world] acme: use http-01 solver"
time="2019-09-08T20:45:13Z" level=debug msg="legolog: [INFO] [do.msj.world] acme: Trying to solve HTTP-01"
time="2019-09-08T20:45:14Z" level=debug msg="No secret name provided" providerName=kubernetescrd
time="2019-09-08T20:45:14Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-09-08T20:45:16Z" level=debug msg="No secret name provided" providerName=kubernetescrd
time="2019-09-08T20:45:16Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2019-09-08T20:45:18Z" level=debug msg="legolog: [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/253187809"
time="2019-09-08T20:45:18Z" level=error msg="Unable to obtain ACME certificate for domains \"do.msj.world\": unable to generate a certificate for the domains [do.msj.world]: acme: Error -> One or more domains had a problem:\n[do.msj.world] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Invalid response from http://do.msj.world/.well-known/acme-challenge/W32sVc-v34BTuUNQFqyNtpjbctrW3fy6K5jttf-MAkk [159.203.156.113]: 503, url: \n" rule="Host(`do.msj.world`) && PathPrefix(`/`)" providerName=default.acme routerName=default/websecure-f35339e15c89b25de38f
I can't pass through that 403 error. Feels like Traefik is not capable of creating the .well-known/acme-challenge inside his container or something ?
Anyone know where did I go wrong ? Or is there some kind of bug involved ?
I've been running in circle for about 10 hours splitted on few days, at this point even a hint would be nice lol
Thanks people!