Traefik v2: Unique deployment labels from the same compose file

Traefik Traefik v2
1

Hi,

We have a situation where we want to deploy multiple stacks at different paths using the same compose file using Traefik v2.

For example:

...
traefik.http.routers.my-app-router.rule: "PathPrefix(`/my-app/${BRANCH_NAME}`)"
traefik.http.routers.my-app-router.middlewares: "my-app-middleware1"
traefik.http.middlewares.my-app-middleware1.stripprefix.prefixes: "/my-app/${BRANCH_NAME}"
traefik.http.routers.my-app-router.service: "my-app-service"
traefik.http.services.my-app-service.loadbalancer.server.port: "8080"
...

That doesn't seem to work, though, since the deployment labels translate to a global configuration in traefik and the two stacks end up clashing.

It isn't possible to use variables in the label keys (i.e. traefik.http.routers.my-app-${BRANCH_NAME}-router.rule). Is there some way to accomplish this declaratively without resorting to adding the labels in a post deployment process using the cli or something similar? Is it possible to scope or namespace the labels per stack?

1 Like
2

Hi @shaun-blake, if the challenge is to factorize -as much as possible- the service definition elements, and only keeping the labels specifics for each instance, you might be interested in https://medium.com/@kinghuang/docker-compose-anchors-aliases-extensions-a1e4105d70bd .

Example on our own's docker-compose stack for Containous slides:

What do you think?

3

I don't think an anchor/alias would help me in this case for the same reason - I'm required to define unique key values for the labels but I can't use a variable in the key. Therefore, I have to manually enumerate every label I wan't to use (which I don't want to do).

Maybe this fuller example will help illustrate the problem better.

docker-compose.yml

version: '3.6'
  services:
    my-service:
      image: my-image:${BRANCH}
    deploy:
      labels:
        traefik.enable: "true"
        traefik.http.routers.my-service-router.rule: "PathPrefix(`/${BRANCH}/my-service`)"
        traefik.http.routers.my-service-router.service: "my-service-lb"
        traefik.http.services.my-service-lb.loadbalancer.server.port: "8080"

I would like to be able to build a my-image:feature1 image then deploy it with docker stack deploy -c docker-compose.yml feature1. Then, I'd like to make some changes on another branch, build a new image called my-image:feature2, and deploy it with docker stack deploy -c docker-compose.yml feature2 without having to touch the docker-compose.yml file. I'd like them to both be deployed at the same time, one at /feature1/my-service and one at /feature2/my-service

The docker-compose.yml file above won't work in this scenario because the label keys aren't unique. I can't see a way to avoid the problem using anchors and aliases. Currently I leave the labels off and, after deploying each stack, I perform a service update to add the labels with label keys that include the branch name to make them unique.

It seems like traefik's docker provider in swarm mode should namespace the configurations with the stack name like how it adds @docker to the configuration names. Or, if there are cases where one swarm stack needs to define a label that another needs to reference, it would be nice if traefik provided a way to do both.

I'm new to Traefik, so I might be misunderstanding something. Does anyone know if what I'd like to do is possible with Traefik already? If not, is there an official place I could create a feature request for it?

4

Hi @shaun-blake, the use case you describe is clear, thanks!

The proposal with the anchors was a solution if your concern would have been reusing as much as possible, but willing to define one compose service per application, which is not the case based on your latest post :slight_smile:

The limitation here is purely related to docker-compose, as the key are immutable. The docker-compose issues about this are all underlying the fact it is wanted, because they expect you to define each service.

Namespacing with Swarm provider might have unwanted impact, since it's not a concept existing in Swarm. With Kubernetes, it would have been different (more complex, but with this kind of abstraction in mind).

Personally, I'm using templating for such cases with docker: I have a token and I apply it with a shell pipe in a one-liner:

docker-compose config docker-compose.yml | sed "s#TOKEN_BRANCH#${BRANCH}#g" \
| docker stack deploy -c - featureX

With TOKEN_BRANCH used in the keys of the labels (or any other key if needed).

5

Hello @shaun-blake, one of my colleagues pointed me to this awesome new feature in Traefik v2 that could totally solve the "router.rule" part of your problem: https://docs.traefik.io/v2.0/providers/docker/#defaultrule .

For example:

version: '3.7'

services:
  reverse-proxy:
    image: traefik:v2.0.0-alpha8
    command:
      - --providers.docker.defaultRule=Host(`localhost`) && Path(`{{ index .Labels "my.path"}}`)
      - --api
    ports:
      - "80:80"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    labels:
      - traefik.enable=false

  # Service reachable at http://localhost/whoami
  webapp:
    image: containous/whoami
    labels:
      - traefik.enable=true
      - my.path=/whoami

If you want to use a TOML static configuration for Traefik (instead of the CLI flag):

# In the traefik.toml, can be done with the CLI also
[providers.docker]
  defaultRule = "Host(`something.com`) && Path(`{{ index .Labels \"my.path\"}}`)"

The only remaining issue is the 8080 port. Just to be sure: are you aware that Traefik tries to automatically detect it: if it is the only exposed port of the backend container, then it is the one used. In my example, the docker image containous/whoami only has the port 80 defined as "exposed" (ref. https://github.com/containous/whoami/blob/master/Dockerfile#L10). So if you are in this case, then your problem is solved with Traefik v2 :slight_smile:

6

Hi @dduportal,

Those were two good ideas you gave. For me, I guess i'd rather just add the labels on after the deployment than have to add a pre-processing step to the compose files. That would certainly work to keep the deployment configuration in the compose file with the rest of the configuration, though.

I thought the defaultRule would work for the majority of our cases, but after experimenting, I ran into two problems.

1. swarmMode

Our deployments are in a swarm so we need to set providers.docker.swarmMode=true,
but when I do, I get an error:

msg="port is missing" container=proxy_webapp-i991rlupiqoybjyvgh1k5f0aa providerName=docker

I tried lots of combinations of parameters to try to get something that would work in a swarm and avoid having to add the traefik.http.services.<xxx>.loadbalancer.server.port label but I didn't find a way.

If I set providers.docker.swarmMode=false, it will start up, but when it attempts to reach a container on another node it gets a timeout and returns a 504.

2. Need for Middleware

The other thing preventing the defaultRule from being very useful for this case is that I almost always need a middleware, too. If I want to proxy a service at /$BRANCH/app, and that service is running at the root path inside the container, I would still need to middleware to route the request properly.

I could attempt to make everything run at a matching path inside the container, but I'd rather not add that as a requirement to every project we deploy. In some cases we're deploying an application that don't provide an easy way to customize the path.

Ideas?

Again, I'm new to Traefik so there might be a way around those two problems I ran into. If you (or anyone else) knows how to avoid them or has other ideas, I'd love to hear them.

Feedback

Ideally something could be done in Traefik to support this use case in a more straight-forward way. It seems like a common use-case in a continuous integration/continuous deployment scenario.

Thanks again!

7

Hi @shaun-blake, you are right, when using swarmMode, the port must be specified, as it's utilizing the Docker Swarm mesh network, and not directly a local container exposing. You might be interested by https://docs.traefik.io/v2.0/providers/docker/#usebindportip.

About your question about the URL rewrite, I strongly advise you to read the documentation page here: https://docs.traefik.io/v2.0/routing/overview/. The defaultRule describes how to match requests in a "Traefik Router". The fact that you want to rewrite the request works as any other reverse-proxy: you have to define that you want to remove the prefix (in nginx it would be a rewrite ^/$BRANCH(/.*)$ $1 last; for example). it's a common pattern, as routing and rewriting request are 2 different tasks that can be different (what about rewriting the path, or not rewriting it).

Thanks for your feedback. Traefik is able to handle your use case out of the box: it's the constraint of using one, and only one docker-compose file, as docker-compose is not a template tool. Have you tried my "command -line" proposal?

Do not forget that docker-compose is not meant to be a template engine, which is the requirement you have here. If you have a CI/CD pipeline, you could totally generate the docker-compose file priori to the deployment, with any template engine, as you will need these files for further management (stop/restart/update) the service(s).

Can you confirm that you can work your way with any of the solutions provided (the command line should be the easier in your case, by adding a step in your CD pipeline)?

8

Hello @dduportal,

A couple of comments on your response:

useBindPortIP

I did see this option already, but, from reading the description I didn't think it would help solve my issue or was related. Did you link to that because it helps with this particular issue somehow?

Middleware

I understand that routing the request like I want requires multiple steps. I only brought it up to point out that, even though the defaultRule let's me avoid needing the router rule in the compose file, I still need the middleware labels to properly route the request, so the defaultRule doesn't solve my issue.

Feedback clarification

While Traefik does work for my scenario (I add the properly named labels after the initial deployment that uses the compose file), the reason for my initial post and my feedback is, that I would prefer it if Traefik provided a way for me to handle this particular scenario in the compose file. The compose file may not be a template but it does support variable substitution in the values and provides a clean, simple, declarative way to see everything related to how a stack is deployed.

I can add the labels in the continuous integration/deployment code, but then it's separated out and isn't as clean and simple anymore.I can also add a pre-processing templating step, but that also adds complication and make things harder outside the context of continuous integration without additional support added. Both options are concessions that I don't want to make if I can avoid them.

Whether Containous wants to put the effort in to let me avoid those concessions is up to you all. I do appreciate you taking the time to try to walk through possible solutions and alternatives to help me get Traefik working the way I want. Thanks!

9

I was thinking about how I could better explain what I'm thinking and maybe this concrete example would help.

I would like to add something like these labels to a service (in this case a compose file):

labels:
  traefik.http.routers.my-router.rule=...
  traefik.http.middlewares.my-middleware.stripprefix.prefixes=...
  traefik.http.services.my-service.loadbalancer.server.port=...

Then, when Traefik (in swarm mode) reads the labels off the service, it could generate the Traefik configuration that would be the equivalent of this:

labels:
  traefik.http.routers.<swarm-service-name>-my-router.rule=...
  traefik.http.routers.<swarm-service-name>-my-router.middlewares="<swarm-service-name>-my-middleware"
  traefik.http.middlewares.<swarm-service-name>-my-middleware.stripprefix.prefixes=...
  traefik.http.routers.<swarm-service-name>-my-router.service="<swarm-service-name>-my-service"
  traefik.http.services.<swarm-service-name>-my-service.loadbalancer.server.port=...

That would solve my issue and my guess is that it would be more intuitive to people (see Traefik Docker label scope). This would also prevent the case where two teams that deploy into the same swarm accidentally choose the same router name and take down the other team's application.

Again, that's just my suggestion, I can work with Traefik as is. Thanks for listening.

Update: I just create a GitHub issue to make this an official feature request - https://github.com/containous/traefik/issues/5134

2 Likes
10

Hi @shaun-blake, thanks for clarifying everything here. My apologies I messed up different subject while trying to help you.

The feature request you opened is clear and clearly convey the intent, thanks a lot for this proposal!

11

No problem, @dduportal. It took me a while to get to something clear and easy to follow. I'll watch the request and see if anything comes of it.

Thanks again for the help and suggestions.

2 Likes
12

@shaun-blake - i have seen your repo with make file to deploy the docker stack.
I have the same pain ( after the migrating to traefikv2 at found this issue ) I thought of following the same way like yours.

Fortunately, found a way to do it .

###The below method works best for me, I have exported the env variables before the stack deploy using Jenkins pipeline
and then mentioned in the docker-compose file.

it works only for this format only ( - "traefik.XXXXX=XXX")

version: "3.7"
services:
whoami:
image: containous/whoami:v1.3.0
networks:
- traefik-public
deploy:
labels:
- "traefik.enable=true"
- "traefik.http.routers.${ROUTER}.rule=Host($DOMAIN.example.com)"
- "traefik.http.routers.{ROUTER}.middlewares=authtraefik" - "traefik.http.routers.{ROUTER}.entrypoints=https"
- "traefik.http.routers.{ROUTER}.tls=true" - "traefik.http.routers.{ROUTER}.tls.certresolver=myresolver"
# Basic Auth
- "traefik.http.middlewares.authtraefik.basicauth.users=user:$$apr1$$q8eZFHjF$$Fvmkk//V6Btlaf2i/ju5n/"
# Swarm Mode
- "traefik.http.services.${ROUTER}.loadbalancer.server.port=80"

networks:
traefik-public:
external: true

13

@vicknesh22,

Hurray! It works! That's exactly what I was looking for. I still think it would be great and intuitive if the labels defined on a service weren't global, so I'll keep the GitHub issue I created open, but this solves the most annoying (maybe too strong of a word) part of integrating with our dynamic deployments.

Thanks for taking the time to post how to do it!