Traefik v1.7 to v2.9 migration for Kubernetes provider

christmascookie · March 10, 2023, 9:36pm

Hello,
We were running v1.7.26 of Traefik as a sidecar container to our main server in Kubernetes.
Our application stack is organized as follows:

Main application pod: 2 containers (a Java gRPC backend container and Traefik sidecar)
Multiple microservices pods: each pod contains 1 container (Python gRPC backend containers).

The overall flow of any request coming in is as follows:

Clients reach the Java gRPC server with request parameters that help identify the microservice to reach.
The Java gRPC application uses the localhost:12000 (http entrypoint, as defined in the config file below) to reach the microservice by using the correct authority by setting the Kubernetes ingress name.
Traffic routed through Traefik reaches the Python gRPC microservice.

We defined custom ingress names for microservices deployed in the same cluster and used Traefik as a reverse-proxy to automatically keep track of them. Our Traefik v1 static file configuration looked as follows:

  defaultEntryPoints = ["http"]

    [entryPoints]
      [entryPoints.http]
      address = ":12000"

      [entryPoints.api]
      address = ":13000"

      [entryPoints.ping]
      address = ":14000"

      [entryPoints.metrics]
      address = ":15000"

    [file]
    directory = "/config/dynamic"
    watch = true

    [kubernetes]
    ingressClass = "custom-ingress-name"

    [ping]
    entryPoint = "ping"

    [api]
    entryPoint = "api"

    [metrics]
      [metrics.prometheus]
      entryPoint = "metrics"

We're in the process of upgrading our Traefik deployment to the latest v2.9.8. We're following the v1-to-v2 migration doc and even used the traefik-migration-tool to bootstrap our new configuration. After a few experimentations, we landed on this new v2 configuration file:

   [entryPoints]
      [entryPoints.http]
        address = ":12000"
      [entryPoints.api]
        address = ":13000"
      [entryPoints.metrics]
        address = ":15000"
      [entryPoints.ping]
        address = ":14000"

    [providers]
      providersThrottleDuration = "2s"
      [providers.kubernetesIngress]
        ingressClass = "custom-ingress-name"
        throttleDuration = "0s"
      [providers.file]
        directory = "/config/dynamic"
        watch = true

    [metrics]
      [metrics.prometheus]
        entryPoint = "metrics"

    [ping]
      entryPoint = "ping"
      terminatingStatusCode = 0

    [api]
      dashboard = true
      insecure = true

    [log]
      level = "DEBUG"

Using this configuration, the Traefik container starts successfully without any errors. The debug logs obtained with this config are as follows:

time="2023-03-10T21:10:36Z" level=info msg="Configuration loaded from file: /config/traefik.toml"
time="2023-03-10T21:10:36Z" level=info msg="Traefik version 2.9.8 built on 2023-02-15T15:23:25Z"
time="2023-03-10T21:10:36Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"api\":{\"address\":\":13000\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"http\":{\"address\":\":12000\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"metrics\":{\"address\":\":15000\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"ping\":{\"address\":\":14000\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"traefik\":{\"address\":\":8080\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}}},\"providers\":{\"providersThrottleDuration\":\"2s\",\"file\":{\"directory\":\"/config/dynamic\",\"watch\":true},\"kubernetesIngress\":{\"ingressClass\":\"custom-ingress-name\"}},\"api\":{\"insecure\":true,\"dashboard\":true},\"metrics\":{\"prometheus\":{\"buckets\":[0.1,0.3,1.2,5],\"addEntryPointsLabels\":true,\"addServicesLabels\":true,\"entryPoint\":\"metrics\"}},\"ping\":{\"entryPoint\":\"ping\"},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"}}"
time="2023-03-10T21:10:36Z" level=info msg="Starting provider aggregator aggregator.ProviderAggregator"
time="2023-03-10T21:10:36Z" level=debug msg="Starting TCP Server" entryPointName=ping
time="2023-03-10T21:10:36Z" level=debug msg="Starting TCP Server" entryPointName=traefik
time="2023-03-10T21:10:36Z" level=debug msg="Starting TCP Server" entryPointName=api
time="2023-03-10T21:10:36Z" level=debug msg="Starting TCP Server" entryPointName=http
time="2023-03-10T21:10:36Z" level=debug msg="Starting TCP Server" entryPointName=metrics
time="2023-03-10T21:10:36Z" level=info msg="Starting provider *file.Provider"
time="2023-03-10T21:10:36Z" level=debug msg="*file.Provider provider configuration: {\"directory\":\"/config/dynamic\",\"watch\":true}"
time="2023-03-10T21:10:36Z" level=info msg="Starting provider *traefik.Provider"
time="2023-03-10T21:10:36Z" level=debug msg="*traefik.Provider provider configuration: {}"
time="2023-03-10T21:10:36Z" level=info msg="Starting provider *ingress.Provider"
time="2023-03-10T21:10:36Z" level=debug msg="*ingress.Provider provider configuration: {\"ingressClass\":\"custom-ingress-name\"}"
time="2023-03-10T21:10:36Z" level=info msg="Starting provider *acme.ChallengeTLSALPN"
time="2023-03-10T21:10:36Z" level=debug msg="*acme.ChallengeTLSALPN provider configuration: {}"
time="2023-03-10T21:10:36Z" level=info msg="ingress label selector is: \"\"" providerName=kubernetes
time="2023-03-10T21:10:36Z" level=info msg="Creating in-cluster Provider client" providerName=kubernetes
time="2023-03-10T21:10:36Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"api\":{\"entryPoints\":[\"traefik\"],\"service\":\"api@internal\",\"rule\":\"PathPrefix(`/api`)\",\"priority\":2147483646},\"dashboard\":{\"entryPoints\":[\"traefik\"],\"middlewares\":[\"dashboard_redirect@internal\",\"dashboard_stripprefix@internal\"],\"service\":\"dashboard@internal\",\"rule\":\"PathPrefix(`/`)\",\"priority\":2147483645},\"ping\":{\"entryPoints\":[\"ping\"],\"service\":\"ping@internal\",\"rule\":\"PathPrefix(`/ping`)\",\"priority\":2147483647},\"prometheus\":{\"entryPoints\":[\"metrics\"],\"service\":\"prometheus@internal\",\"rule\":\"PathPrefix(`/metrics`)\",\"priority\":2147483647}},\"services\":{\"api\":{},\"dashboard\":{},\"noop\":{},\"ping\":{},\"prometheus\":{}},\"middlewares\":{\"dashboard_redirect\":{\"redirectRegex\":{\"regex\":\"^(http:\\\\/\\\\/(\\\\[[\\\\w:.]+\\\\]|[\\\\w\\\\._-]+)(:\\\\d+)?)\\\\/$\",\"replacement\":\"${1}/dashboard/\",\"permanent\":true}},\"dashboard_stripprefix\":{\"stripPrefix\":{\"prefixes\":[\"/dashboard/\",\"/dashboard\"]}}},\"serversTransports\":{\"default\":{\"maxIdleConnsPerHost\":200}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=internal
time="2023-03-10T21:10:36Z" level=debug msg="Configuration received: {\"http\":{},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=file
time="2023-03-10T21:10:36Z" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
time="2023-03-10T21:10:36Z" level=debug msg="Added outgoing tracing middleware ping@internal" entryPointName=ping routerName=ping@internal middlewareName=tracing middlewareType=TracingForwarder
time="2023-03-10T21:10:36Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=ping middlewareName=traefik-internal-recovery
time="2023-03-10T21:10:36Z" level=debug msg="Added outgoing tracing middleware prometheus@internal" middlewareType=TracingForwarder entryPointName=metrics routerName=prometheus@internal middlewareName=tracing
time="2023-03-10T21:10:36Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=metrics
time="2023-03-10T21:10:36Z" level=debug msg="Added outgoing tracing middleware api@internal" middlewareName=tracing middlewareType=TracingForwarder entryPointName=traefik routerName=api@internal
time="2023-03-10T21:10:36Z" level=debug msg="Added outgoing tracing middleware dashboard@internal" entryPointName=traefik routerName=dashboard@internal middlewareName=tracing middlewareType=TracingForwarder
time="2023-03-10T21:10:36Z" level=debug msg="Creating middleware" routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix entryPointName=traefik
time="2023-03-10T21:10:36Z" level=debug msg="Adding tracing to middleware" middlewareName=dashboard_stripprefix@internal entryPointName=traefik routerName=dashboard@internal
time="2023-03-10T21:10:36Z" level=debug msg="Creating middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex
time="2023-03-10T21:10:36Z" level=debug msg="Setting up redirection from ^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$ to ${1}/dashboard/" middlewareType=RedirectRegex entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal
time="2023-03-10T21:10:36Z" level=debug msg="Adding tracing to middleware" middlewareName=dashboard_redirect@internal entryPointName=traefik routerName=dashboard@internal
time="2023-03-10T21:10:36Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareType=Recovery middlewareName=traefik-internal-recovery
time="2023-03-10T21:10:36Z" level=debug msg="Creating middleware" entryPointName=ping middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-03-10T21:10:36Z" level=debug msg="Creating middleware" middlewareType=Metrics entryPointName=traefik middlewareName=metrics-entrypoint
time="2023-03-10T21:10:36Z" level=debug msg="Creating middleware" middlewareType=Metrics entryPointName=api middlewareName=metrics-entrypoint
time="2023-03-10T21:10:36Z" level=debug msg="Creating middleware" entryPointName=http middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-03-10T21:10:36Z" level=debug msg="Creating middleware" entryPointName=metrics middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-03-10T21:10:36Z" level=debug msg="Creating middleware" entryPointName=ping middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-03-10T21:10:36Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-03-10T21:10:36Z" level=debug msg="Creating middleware" entryPointName=api middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-03-10T21:10:36Z" level=debug msg="Creating middleware" middlewareType=Metrics entryPointName=http middlewareName=metrics-entrypoint
time="2023-03-10T21:10:36Z" level=debug msg="Creating middleware" entryPointName=metrics middlewareName=metrics-entrypoint middlewareType=Metrics
time="2023-03-10T21:10:36Z" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"namespace-java-backend-echomodel-default-echomodel-customingressname\":{\"service\":\"namespace-java-backend-echomodel-model\",\"rule\":\"Host(`default.echomodel.customingressname`) \\u0026\\u0026 PathPrefix(`/`)\"},\"abc-microservice-abc-microservice-default-abc-microservice-customingressname\":{\"service\":\"abc-microservice-abc-microservice-model\",\"rule\":\"Host(`default.abc-microservice.customingressname`) \\u0026\\u0026 PathPrefix(`/`)\"}},\"services\":{\"namespace-java-backend-echomodel-model\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://<server-ip>:9000\"}],\"passHostHeader\":true}},\"abc-microservice-abc-microservice-model\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://<microservice-ip>:50051\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"udp\":{}}" providerName=kubernetes
time="2023-03-10T21:10:36Z" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
....
....

Please see the attached traefik_log file for the full container log: traefik_log - Pastebin.com

In the older deployments, the following grpcurl command (to simply list the underlying services) gives us the correct response.

grpcurl -plaintext -authority "custom.kubernetesingress.name" localhost:12000 list

However, in our new deployments using the v2.9.8 Traefik containers, the same gRPC commands (and requests generally used to generate valid responses) don't seem to get forwarded to the Python microservices. We checked the application logs and there's no trace of any incoming requests.

Could you please tell us if our new v2 configuration is migrated correctly to v2 standards? Since Traefik can correctly detect the sample microservices we deployed, there seems to be general networking connectivity between all the pods. Could you please point us to any tips to help debug why the gRPC commands are not forwarded to the microservices?
One of the hypotheses we had was that moving from v1 to v2, the defaultEntrypoints config has been deprecated. Also we think this log is pertinent No entryPoint defined for this router, using the default one(s) instead: [api http metrics ping]. Could this be the reason? If so, is there a way to set the http entrypoint as a default entrypoint in v2? Or could you please point us to any workarounds?

Thank you for your help! We'd really appreciate any insights on debugging this issue.

svx · March 13, 2023, 2:02pm

Hi @christmascookie.
Thanks for your interest in Traefik!

Could you post your full configuration?
Could you do a request to /api/rawdata and post the output? (That returns information about dynamic configurations, errors, status, and dependency relations.)

Topic		Replies	Views
Traefik not respecting entrypoint constraints Traefik v2 kubernetes-crd , tcp	0	518	September 18, 2019
gRPC Setup for K3s Traefik v2 kubernetes-ingress	1	607	August 11, 2023
Unable to connect to traefik dashboard Traefik v2 file , kubernetes-ingress	1	661	September 20, 2022
Global redirect and global default entrypoints Traefik v2 kubernetes-ingress	1	1034	September 23, 2019
Adding entry point, different procedure than v2? Traefik v3 (latest) kubernetes-crd , kubernetes-ingress	0	163	May 15, 2024

Traefik v1.7 to v2.9 migration for Kubernetes provider

Related topics