With Helm chart version 35.0.0 and Traefik version 3.3.5, I'm trying to set up the Helm chart in Kubernetes to get a Let's Encrypt TLS/HTTPS certificate and use it for an IngressRoute, but whenever I add the Proxy Protocol stuff so that the LoadBalancer can preserve client IP addresses as per Civo's docs (the trustedIPs, annotations, and externalTrafficPolicy), it stops working.
I don't get a certificate, and I saw this error in the logs:
2025-04-23T16:30:48Z DBG github.com/go-acme/lego/v4@v4.22.2/log/logger.go:48 > [INFO] Deactivating auth: https ://acme-staging-v02.api.letsencrypt.org/acme/authz/196501024/16965213324 lib=lego
2025-04-22T13:09:33Z ERR Unable to obtain ACME certificate for domains error="unable to generate a certificate for the domains [dashboard.example.blue]: error: one or more domains had a problem:\n[dashboard.example.blue] inv.alid authorization: acme: error: 400 :: urn:ietf:params:acme:error:connection :: 74.220.25.170: Fetching http ://dashboard.example.blue/.well-known/acme-challenge/DN4ggaxOm7FlVZiHQqiGe4-x9lN1EJkHA5n6TymfkJ4: Error getting validation data\n" ACME CA=https ://acme-staging-v02.api.letsencrypt.org/directory acmeCA=https ://acme-staging-v02.api.letsencrypt.org/directory domains=["dashboard.example.blue"] providerName=staging.acme routerName=websecure-dash-drupal-42da837a8cc7a01b7ea1@kubernetescrd rule=Host(dashboard.example.blue)
Also:
% curl -I https://dashboard.example.blue
curl: (35) error:0A000126:SSL routines::unexpected eof while reading
% curl -I http://dashboard.example.blue
curl: (52) Empty reply from server
Here's the Terraform resource for the chart:
resource "helm_release" "traefik" {
name = "traefik"
namespace = kubernetes_namespace.drupal_dashboard.metadata[0].name
repository = "https://traefik.github.io/charts"
chart = "traefik"
version = var.traefik_helm_chart_version
values = [
yamlencode({
additionalArguments = [
"--entryPoints.web.address=:${var.http_port}",
"--entryPoints.web.proxyProtocol.trustedIPs=${join(",", ["0.0.0.0/0"])}",
"--entryPoints.websecure.address=:${var.https_port}",
"--entryPoints.websecure.proxyProtocol.trustedIPs=${join(",", ["0.0.0.0/0"])}"
]
service = {
annotations = {
"kubernetes.civo.com/loadbalancer-enable-proxy-protocol" = "send-proxy-v2"
"kubernetes.civo.com/firewall-id" = var.firewall_id_annotation_value
}
spec = {
externalTrafficPolicy = "Local"
}
}
certificatesResolvers = {
(var.letsencrypt_staging_environment_name) = {
acme = {
caServer = "https://acme-staging-v02.api.letsencrypt.org/directory"
email = var.technical_contact_email
storage = local.tls_certificate_data_path
httpChallenge = {
entryPoint = "web"
}
}
}
(var.letsencrypt_production_environment_name) = {
acme = {
caServer = "https://acme-v02.api.letsencrypt.org/directory"
email = var.technical_contact_email
storage = local.tls_certificate_data_path
httpChallenge = {
entryPoint = "web"
}
}
}
}
persistence = {
enabled = true
storageClass = var.acme_storage_class
path = var.tls_certificate_data_directory
}
# Give the application user write access to the data file. See:
# https://github.com/traefik/traefik-helm-chart/issues/396#issuecomment-1873454777
podSecurityContext = {
fsGroup = var.traefik_user_id
fsGroupChangePolicy = "OnRootMismatch"
runAsGroup = var.traefik_user_id
runAsNonRoot = true
runAsUser = var.traefik_user_id
}
deployment = {
initContainers = [
{
name = "volume-permissions"
image = "busybox:latest"
command = [
"sh",
"-c",
"ls -alF /; touch ${local.tls_certificate_data_path}; chmod -v 600 ${local.tls_certificate_data_path}; ls -alF ${local.tls_certificate_data_path}"
]
securityContext = {
runAsNonRoot = true
runAsGroup = var.traefik_user_id
runAsUser = var.traefik_user_id
}
volumeMounts = [
{
name = "data"
mountPath = var.tls_certificate_data_directory
}
]
}
]
}
})
]
}
I'm quite sure that the storage file / persistence is working fine as it gets written to, and the permissions look okay:
~ $ ls -alF /data/acme.json
-rw------- 1 65532 65532 3609 Apr 23 21:44 /data/acme.json
Additional Information
I also tried a couple of things with the same result:
- Both
send-proxy(v1) andsend-proxy-v2(v2) - Splitting the IngressRoute into 2, for
webandwebsecure.
Even with debugging on, I can't get any more error information than Error getting validation data. What's the error? Why can't it get the validation data? I feel like there should be more information here explaining what the problem is, but there isn't. So that's why I'm thinking that this is a bug; it's not telling me that I'm doing something wrong.
Thanks so much for making this project available and maintaining it. I really like the built-in support for Let's Encrypt; that's why I chose Traefik over Nginx. Keep up the great work!