Why I still prefer v1 over v2, or why I really don't like to use v2

Traefik Traefik v2
,
1

Hello everyone,

as suggested / requested multiple times by the @traefik account and @patricia_dugan on Twitter, I am now taking this topic to the community forums.

First of all I must say that I have now migrated to v2, but it took me quite some time and this post will focus on the reasons for that. This is meant as constructive and functional feedback to the developers and can also serve as examples for the community. I will list positive and negative aspects of my migration story. Let's start with some background on my setup.

My environment
I am using traefik as a Docker service pinned to individual Docker nodes in a Docker Swarm cluster. Meaning, I am not using the ingress or loadbalancing features, because I want specific sites to only be reachable on specific nodes / IP addresses and therefore I also do not run traefik with Swarm awareness. So my 4 traefik instances are basically single instances observing and handling traffic for a single Docker node each. Before you ask: the Docker Swarm cluster is used for other purposes than loadbalancing the web traffic, like service management via Docker Stck. Swarm aside, I think my deployment scenario is pretty straight forward and similar to a single traefik instance serving a single Docker node, just 4 time repeated.

My deployment story
I deploy all my Docker nodes, including web application / services and of course traefik with Ansible on the Swarm cluster via Docker Stack definitions. This means I can use Ansible Jinja2 templating to generate the required YAML configuration files for traefik and also the Docker Stack definitions. Besides the general traefik configuration being done via CLI parameters, all web application / services are served with traefik via the Docker provider based on container (not Swarm service) labels.

My migration story
The original Docker Stack definition for my traefik v1 stack basically looked like this in the Ansible deployment task:

- name: deploy traefik stack
  command: docker stack deploy --prune --compose-file - traefik-{{ inventory_hostname_short }}
  args:
    stdin: |
      version: '3.7'
      services:
        traefik:
          image: traefik:1.7.10
          init: true
          command:
            - "--api"
            - "--api.entryPoint=traefik"
            - "--docker"
            - "--docker.domain=example.com"
            - "--docker.exposedByDefault=false"
            - "--docker.templateVersion=2"
            - "--docker.filename=/etc/traefik/docker.tmpl"
            - "--docker.watch=true"
            - "--acme"
            - "--acme.email=..."
            - "--acme.entryPoint=https"
            - "--acme.storage=/etc/traefik/acme/acme.json"
            - "--acme.onHostRule=true"
            - "--acme.onDemand=false"
            - "--acme.httpChallenge.entryPoint=http"
            - "--entryPoints=Name:http Address::80 Redirect.EntryPoint:https"
            - "--entryPoints=Name:https Address::443 TLS"
            - "--entryPoints=Name:traefik Address::8080"
            - "--defaultentrypoints=http,https"
            - "--accessLog"
            - "--accessLog.filePath=/var/log/traefik/access.log"
            - "--metrics.prometheus"
            - "--metrics.prometheus.entryPoint=traefik"
            - "--tracing.jaeger.samplingServerURL=http://agent:5778/sampling"
            - "--tracing.jaeger.localAgentHostPort=agent:6831"
          logging:
            driver: "journald"
          labels:
            traefik.enable: "true"
            traefik.backend: "traefik"
            traefik.frontend.rule: "Host:{{ inventory_hostname }}"
            traefik.frontend.auth.basic: "..."
            traefik.port: "8080"
            traefik.docker.network: "traefik-{{ inventory_hostname_short }}_dmz"
          ports:
            - target: 80
              published: 80
              protocol: tcp
              mode: host
            - target: 443
              published: 443
              protocol: tcp
              mode: host
          networks:
            - dmz
          environment:
            DOCKER_TEMPLATE_CHECKSUM: "{{ docker_template.checksum }}"
          volumes:
            - /home/docker-traefik/docker.tmpl:/etc/traefik/docker.tmpl
            - /home/docker-traefik/acme:/etc/traefik/acme
            - /home/docker-traefik/logs:/var/log/traefik
            - /var/run/docker.sock:/var/run/docker.sock
            - /dev/null:/traefik.toml
          deploy:
            endpoint_mode: dnsrr
            placement:
              constraints:
                - node.hostname == {{ inventory_hostname }}
      networks:
        dmz:
          driver: overlay
          attachable: true
          ipam:
            driver: default
            config:
              - subnet: "10.8.{{ groups.docker.index(inventory_hostname) }}.0/24"
  delegate_to: "{{ groups['docker_swarm_manager'][0] }}"
  delegate_facts: True
  become: yes
  tags: docker-traefik

As you can see this was a pretty straightforward setup with the following traefik features being used:

  • Docker provider (with custom template to generate static backend names)
  • Certificates via Let's Encrypt (ACME)
  • Redirect from HTTP to HTTPS for all containers
  • Default EntryPoints for HTTP and HTTPS
  • EntryPoint traefik to be used for Dashboard / API interface (served with authentication via container labels)
  • Access logging, Prometheus metrics and Jeager tracing

Now to convert this for use with traefik v2, it now looks like the following:

- name: deploy traefik stack
  command: docker stack deploy --prune --compose-file - traefik-{{ inventory_hostname_short }}
  args:
    stdin: |
      version: '3.7'
      services:
        traefik:
          image: traefik:2.0
          init: true
          command:
            - "--api"
            - "--api.insecure=true"
            - "--api.dashboard=true"
            - "--providers.docker"
            - "--providers.docker.exposedByDefault=false"
            #- "--docker.templateVersion=2"
            #- "--docker.filename=/etc/traefik/docker.tmpl"
            - "--providers.docker.watch"
            - "--providers.file"
            - "--providers.file.filename=/etc/traefik/file.yaml"
            - "--providers.file.watch"
            #- "--acme"
            - "--certificatesResolvers.letsencrypt.acme.email=..."
            #- "--acme.entryPoint=https"
            - "--certificatesResolvers.letsencrypt.acme.storage=/etc/traefik/acme/acme.json"
            - "--certificatesResolvers.letsencrypt.acme.tlsChallenge=true"
            #- "--acme.onHostRule=true"
            #- "--acme.onDemand=false"
            #- "--acme.httpChallenge.entryPoint=http"
            - "--entryPoints.http"
            - "--entryPoints.http.address=:80"
            #- "-- Redirect.EntryPoint:https"
            - "--entryPoints.https"
            - "--entryPoints.https.address=:443"
            #- "-- TLS"
            - "--entryPoints.traefik.address=:8080"
            #- "--defaultentrypoints=http,https"
            - "--accessLog"
            - "--accessLog.filePath=/var/log/traefik/access.log"
            - "--metrics.prometheus=true"
            - "--metrics.prometheus.entryPoint=traefik"
            - "--tracing.jaeger=true"
            - "--tracing.jaeger.samplingServerURL=http://agent:5778/sampling"
            - "--tracing.jaeger.localAgentHostPort=agent:6831"
          logging:
            driver: "journald"
          labels:
            traefik.enable: "true"
            traefik.docker.network: "traefik-{{ inventory_hostname_short }}_dmz"
            traefik.http.routers.traefik-https.rule: &rule "Host(`{{ inventory_hostname }}`)"
            traefik.http.routers.traefik-https.entrypoints: "https"
            traefik.http.routers.traefik-https.middlewares: "auth@file"
            traefik.http.routers.traefik-https.tls: "true"
            traefik.http.routers.traefik-https.tls.certResolver: "letsencrypt"
            traefik.http.routers.traefik-http.rule: *rule 
            traefik.http.routers.traefik-http.entrypoints: "http"
            traefik.http.routers.traefik-http.middlewares: "https@file"
            traefik.http.services.traefik.loadbalancer.server.port: "8080"
          ports:
            - target: 80
              published: 80
              protocol: tcp
              mode: host
            - target: 443
              published: 443
              protocol: tcp
              mode: host
          networks:
            - dmz
          environment:
            FILE_CONFIG_CHECKSUM: "{{ file_config.checksum }}"
          volumes:
            #- /home/docker-traefik/docker.tmpl:/etc/traefik/docker.tmpl
            - /home/docker-traefik/file.yaml:/etc/traefik/file.yaml
            - /home/docker-traefik/acme:/etc/traefik/acme
            - /home/docker-traefik/logs:/var/log/traefik
            - /var/run/docker.sock:/var/run/docker.sock
            #- /dev/null:/traefik.toml
          deploy:
            endpoint_mode: dnsrr
            placement:
              constraints:
                - node.hostname == {{ inventory_hostname }}
      networks:
        dmz:
          driver: overlay
          attachable: true
          ipam:
            driver: default
            config:
              - subnet: "10.8.{{ groups.docker.index(inventory_hostname) }}.0/24"
  delegate_to: "{{ groups['docker_swarm_manager'][0] }}"
  delegate_facts: True
  become: yes
  tags: docker-traefik

As you can see I was able to keep most of the general configuration in the CLI parameters.

  • Docker provider (no need for a custom template, because backend (service) names are now static by design via container labels) :+1:
  • (New) File provider for reusable middlewares, e.g. for authentication and redirects :+1:
  • Certificates (still) via Let's Encrypt (ACME, now via TLS challenge), but unfortunately onHostRule is gone :-1: (this will lead to some serious redundant container labels)
  • EntryPoints for HTTP and HTTPS (the "Default" part is gone :-1:)
  • EntryPoint traefik to be used for Dashboard / API interface (served with authentication via container labels) (still there :slight_smile:)
  • Access logging, Prometheus metrics and Jeager tracing (still there :slight_smile:)

IMHO completely missing are the following features:

  • Being able to set sensible defaults for EntryPoints, ACME and TLS configuration (see :-1: above)
  • Redirect from HTTP to HTTPS for all containers :-1:

Of course traefik v2 still provides some new stuff, like the following:

  • Static middleware, router and service names (see :+1: above, although this is a side effect of everything needing to be configured explicitly instead of general defaults being applied / inherited)
  • More flexibility by being able to configure the above for each container individually, especially HTTPS redirection and TLS configuration :+1:
  • More flexibility by allowing for more complex rules on routers via new syntax (while breaking the existing host rule syntax) :upside_down_face:
  • Serving and routing TCP traefik including TLS :star_struck:

This means I think I totally understand why the shift was from having the default EntryPoints, HTTPS redirection, ACME and TLS configuration on the general level to the router level. The only thing I am really missing here is: why not have the best of both worlds? Being able to specify sensible defaults for EntryPoints, HTTPS redirection, ACME and TLS configuration on the general level, while still being able to override the defaults on a per router basis?

You can already imagine the potential configuration overhead (duplication instead of DRY) this change has for me by looking at the traefik container labels on the 2 traefik containers.

Comparison of v1 and v2 container labels

Example of serving a buildbot instance which is providing different ports for different access levels:

Before (with v1.7):

labels:
    traefik.enable: "true"
    traefik.user.backend: "buildbot-user"
    traefik.user.frontend.rule: "Host:buildbot.example.com;Method:GET,HEAD"
    traefik.user.port: "8080"
    traefik.github.backend: "buildbot-github"
    traefik.github.frontend.rule: "Host:buildbot.example.com;Method:POST;Path:/change_hook/github"
    traefik.github.port: "8080"
    traefik.admin.backend: "buildbot-admin"
    traefik.admin.frontend.rule: "Host:buildbot.example.com;PathPrefixStrip:/admin/"
    traefik.admin.frontend.auth.basic: "..."
    traefik.admin.port: "8443"
    traefik.docker.network: "traefik-{{ inventory_hostname_short }}_dmz"

After (with v2.0):

labels:
    traefik.enable: "true"
    traefik.docker.network: "traefik-{{ inventory_hostname_short }}_dmz"
    traefik.http.middlewares.buildbot-admin.stripprefix.prefixes: "/admin"
    traefik.http.routers.buildbot-user-https.rule: &rule "Host(`buildbot.example.com`) && (Method(`GET`,`HEAD`) || (Method(`POST`) && Path(`/change_hook/github`)))"
    traefik.http.routers.buildbot-user-https.entrypoints: "https"
    traefik.http.routers.buildbot-user-https.service: &service "buildbot-user@docker"
    traefik.http.routers.buildbot-user-https.tls: "true"
    traefik.http.routers.buildbot-user-https.tls.certResolver: "letsencrypt"
    traefik.http.routers.buildbot-user-http.rule: *rule 
    traefik.http.routers.buildbot-user-http.entrypoints: "http"
    traefik.http.routers.buildbot-user-http.middlewares: "https@file"
    traefik.http.routers.buildbot-user-http.service: *service
    traefik.http.routers.buildbot-admin-https.rule: &rule "Host(`buildbot.example.com`) && PathPrefix(`/admin/`)"
    traefik.http.routers.buildbot-admin-https.priority: 200
    traefik.http.routers.buildbot-admin-https.entrypoints: "https"
    traefik.http.routers.buildbot-admin-https.middlewares: "auth@file,buildbot-admin@docker"
    traefik.http.routers.buildbot-admin-https.service: &service "buildbot-admin@docker"
    traefik.http.routers.buildbot-admin-https.tls: "true"
    traefik.http.routers.buildbot-admin-https.tls.certResolver: "letsencrypt"
    traefik.http.routers.buildbot-admin-http.rule: *rule 
    traefik.http.routers.buildbot-admin-http.entrypoints: "http"
    traefik.http.routers.buildbot-admin-http.middlewares: "https@file"
    traefik.http.routers.buildbot-admin-http.service: *service
    traefik.http.services.buildbot-user.loadbalancer.server.port: "8080"
    traefik.http.services.buildbot-admin.loadbalancer.server.port: "8443"

As you can see I am already making use of YAML anchors and aliases for DRY (thanks to @sudo_bmitch on Twitter), but the number of labels still went up from 11 to 25. And this is after I already merged the "github" backend into the "user" service by using a more complex router rule. I am also missing the simplicity and efficiency of being able to match something and modify it at the same time, e.g. the rule matcher and modifier PathStripPrefix.

My conclusion
I would really like to ask and urge the developers of traefik v2 to make it possible again to specify some defaults like HTTPS, ACME, TLS for routers on a general level to be inherited unless overridden. Having 2 routers for every service, just because you want TLS feels really redundant, especially if you have to specify the certificateResolver over and over while only one is configured (e.g. Let's Encrypt). Having to specify a middleware on every HTTP router to redirect to HTTPS feels again really redundant.

I am looking forward to the discussion and feedback around the aspects I brought up. I hope that I have made my points clear. In case I missed something or did something wrong and you have an idea on how to fix it or make it even less verbose and apply DRY, please feel free to comment here. Thanks in advance!

Best regards,
Marc

P.S.: The stupid rule of prohibiting new users from posting no more than 2 links in a post just made me remove all the references I collected for this. This means it's @traefik fault that this post has no references attached to it.

4 Likes
2

To add some more background information why the issues mentioned above are so relevant for me: I am serving 38 docker containers with traefik in a very similar fashion to the example above, so the increase of redundancy in label-based configuration had a huge impact on the simplicity and compactness of my overall environment.

1 Like
3

Hello,

thanks for your feedback.

We recommend to not use the traefik entry point to route the Dashboard / API, we create api@internal and the insecure option to have better and secure routing.

You can already create a global HTTP to HTTPS redirection: (With that pattern you can remove at least 50% of your labels)

the onHostRule static option has been removed to allow mainly to use dynamic wildcard certificates (It was a very requested feature).

Some proposals are already opened on your main points:

This "stupid" rule is made to avoid spam.

1 Like
4

Hello @ldez,

thank you very much for your response. Sorry for my late response, but I am quite busy at the moment.

Thanks, I already implemented this change.

I don't think this is a good idea, because this will serve non-existing HTTP services, e.g. non-existing DNS entries (if requested via a modified Host-header). I only want to have the HTTP to HTTPS redirection for the existing services and rules.

These look promising, I hope a solution will be implemented soon.

I don't think this really helps to avoid spam, since spam can already happen with just 1 or 2 links. BTW-- I cannot even completely quote the sections of your reply above, because the number of links is still limited.

5

A redirection by Traefik v1 on an entrypoint, do the same thing, because the redirection on the entrypoint don't care about the request information (Host, Headers, ...)

But in the v2 and with the global redirection, you can put the list of your domains instead of the simple catch all, so you can create very precise redirection.

Since your first post I increased the limitation to 4 links for new users. The limitation is just the default configuration of Discource to avoid spam.

6

Thanks, for my lab I did upgrade to 2.0, but there was no significant need for me to upgrade the old 1.7 stack which is working nicely already. The migration would mean redoing the labels and risk major downtime.

There's still a few glitches that need to be addressed in 2.x such as the api dashboard. Plus the forcing of people to go to the EE version if they want HA. I've done rolling updates using docker stack deploy of Traefik with the consul successfully so far at least there was no noticeable downtime (i.e. uptime robot didn't complain).

So there's no immediate drive to upgrade. Generally I would follow that rule where if there's no actual business value or development value don't bother.

As far as business value goes, because of the TLS forwarding, we can reduce our EIP and swarm count by merging two swarms and having it route "intranet" requests through an internal traefik. Though it costs less on paper (and I've implemented that on my lab) it is not really worth having the risk of mixing two vastly different concerns because the cost of the machines is cheaper than human cost.

Like I said, I just did it for my home lab, which is running off a cable modem so I only get one external IP. To prove it can be done.

Some tips I can share is to use the "chain" so you can reduce the work. that's what I did in Trajano base Docker swarm stacks

7

@ldez Are there any plans to improve the situation regarding boilerplate middlewares just to strip a path that has already been matched? Since PathStripPrefix is not available in v2, it now takes more than just a simple rule to mount an application under another path. Thanks in advance!

8

The separation between routing and transformations is a design choice, so we will not create routing rule with transformations.

9

@ldez Thank you very much for the feedback so far. Merry Christmas!

10

Hey Marc @mback2k ,

Thanks for your honest feedback and the time you took to detail your journey in the transition from Traefik V1 to V2.

Indeed, doing this migration is not something we do while checking our Instagram feed. It's still a serious technical task. We need to tackle each feature required in our stack one at a time. I was in your shoes as I was using Traefik V1 for two years.

To conclude, is it fair to say that @Idez addressed all your thumbs down (I saw four)?

  • Certificates (still) via Let's Encrypt (ACME, now via TLS challenge), but unfortunately onHostRule is gone :-1: (this will lead to some serious redundant container labels)
  • EntryPoints for HTTP and HTTPS (the "Default" part is gone :-1:)
  • Being able to set sensible defaults for EntryPoints, ACME and TLS configuration (see :-1: above)
  • Redirect from HTTP to HTTPS for all containers :-1:

To me, Traefik V2 addresses all these requirements at the end.

Let me know if I missed something!

11

@pascalandy What is traefik 2 alternative to

--entryPoints=Name:http Address::80 whitelist.sourcerange:10.20.1.16,192.168.11.17,192.168.7.185,192.168.8.185
12

Check out https://docs.traefik.io/v2.1/middlewares/ipwhitelist/
It's a dynamic configuration :slight_smile:

13

@pascalandy, so imagine this scenario:

traefik is used as in ingress controller in a kubernetes cluster. It must be allowed to be only contactable from the load balancer set of IPs, this is how security of the system is designed. No one should be able to call traefik from any other IP than load balancer. With traefik 1 it was working with the example I gave above.

With traefik 2 however, this has to go to IngressRoute of every single application. IngresRoutes are authored by developers and I need to trust them to add the middleware in there for each app.

But the things is I do not trust them. Someone can forget to put it there, or have this idea that they "just test it" without the middleware, leaving the app wide open to whoever decides to connect to the traefik address.

It does not look like the requirement to secure traefik with static configuration is fulfilled by traefik 2 here. What do you think?

PS. it is all possible to overcome with custom admission control, or integrating some more checks in CI/CD pipeline which will check for the middleware which you have to implement yourself, but you did not need any of that with traefik 1 . It just worked.

14

Hi @zespri,

I was reading with interest your last post and I think you've pointed out a limitation, that until the new v2.2 it was not possible to solve.
With 2.2, again, a "default" ipWhitelist middleware can be applied directly to any entryPoint from the static configuration :

---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: traefik
  labels:
    app: traefik

spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik
  template:
    metadata:
      labels:
        app: traefik
    spec:
      serviceAccountName: traefik-ingress-controller
      containers:
        - name: traefik
          image: traefik:v2.2.0-rc4
          args:
            - --log.level=DEBUG
            - --api
            - --api.insecure
            - --entrypoints.web.address=:80
            - --entrypoints.web.http.middlewares=ipwl@file
            - --providers.kubernetescrd
            - --providers.file
            - --providers.file.filename=/etc/traefik/dynamic.toml
          ports:
            - name: web
              containerPort: 80
            - name: admin
              containerPort: 8080

          volumeMounts:
            - mountPath: /etc/traefik/dynamic.toml
              name: traefik-dynamic-configmap
              subPath: dynamic.toml
              readOnly: true
      volumes:
        - name: traefik-dynamic-configmap
          configMap:
            name: traefik-dynamic-configmap

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: traefik-dynamic-configmap
  namespace: default
data:
  dynamic.toml: |
    [http.middlewares.ipwl.ipWhiteList]
      sourceRange = ["foobar"]
2 Likes
15

Yah with 2.2 I was able to remove a few more labels for the services specifically adding a "default" chain. I am updating my demo swarm with these new changes right now.

16

Here's two more things to add to the reasons.

  1. The TCP Router doesn't have a working example provided databases as alluded to the 2.0 announcement so one of the selling points to migrate doesn't pan out anymore.
  2. They removed the notion of having a default list of entry points so if you had both HTTP and HTTPS with HTTP just being a redirect to HTTPS, you need to explicitly define the HTTPS route.

The only features in 2.x I actually use are

  1. SSL Client certificate verification on a separate HostSNI
  2. TLS Challenge for LetsEncrypt (2.2 feature, but it looks a bit cleaner anyway)
17

@rtribotte looks like a good one. Did not have a chance to try that yet, but it sounds like it should solve the problem. Thank you for posting that!