Traefik is unable to strip prefix before forwarding request to a service

[level0-coder]

Hello and greetings good folks,

I am just learning and trying out traefik for my selfhosting hobbies, and sadly I am running into problems with some routine. I am pretty sure, it's a stupid mistake from my end, but I would like to really know what I am doing wrong and can refocus my mental model about Traefik.

I have two services running inside docker on a raspberrypi, one is traefik itself and the other one is a service called monica.

When I port forward monica's internal 80 port to let's say port 9999 of the host, I can reach the service on http://rpihome:9999. No issues at all.

However, I would like to access the service at http://rpihome/monica. To achieve this I have the following labels on monica's docker container:

"traefik.http.middlewares.monica-stripprefix.stripprefix.prefixes": "/monica",
"traefik.http.routers.monica.rule": "Path(`/monica`)"
"traefik.http.routers.monica.middlewares": "monica-stripprefix",

I have also tried changing the rule from Path to PathPrefix as well as Host() && PathPrefix() without any luck.

I always get a 404 on http://rpihome/monica and my monica container logs indeed give me a log that there was a GET request for /monica which was returned 404.

I didn't even knew that I had to link the middlewares to routers like this "traefik.http.routers.monica.middlewares": "monica-stripprefix",, but was pointed out to this by a kind stranger on Discord.

Can someone shed a light on what's missing? Looking forward.

Regards,
Anubhav.

[bluepuma77]

When you see access log entries on your service, then it seems the labels are being picked up.

Have you tried using Host("example.com") (use backticks!!!) to see if the routing works when using root URL, without stripprefix?

The Traefik stripprefix example sets " differently, not sure if that matters. Also Traefik middleware assignment is usually done with = and it seems with @docker when using Docker labels.

[level0-coder]

Thanks for the hints.

@bluepuma77

Have you tried using Host("example.com") (use backticks!!!) to see if the routing works when using root URL, without stripprefix?

Yes this works on the root of the domain, however my goal would be to have all my services work with a slash, so domain/service1, domain/service2 etc

@bluepuma77

The Traefik stripprefix example sets " differently, not sure if that matters.

I am generating my docker containers by terraform. Here is the code that creates the labels:

resource "docker_container" "monica_container" {
  image = docker_image.monica_image.name
  name  = "monica"
  env = [
    "DB_HOST=mysql"
  ]
  networks_advanced {
    name = docker_network.private_network.name
  }
  labels {
    label = "traefik.http.routers.monica.rule"
    value = "Host(`rpihome`) && Path(`/monica`)"
  }
  labels {
    label = "traefik.http.middlewares.monica-stripprefix.stripprefix.prefixes"
    value = "`/monica`"
  }
  labels {
    label = "traefik.http.routers.monica.middlewares"
    value = "monica-stripprefix@docker"
  }
}

Once the containers were created, I did a docker inspect to find out the labels, so I also suspect that the use of " should not matter so much.

@bluepuma77

Also Traefik middleware assignment is usually done with = and it seems with @docker when using Docker labels.

I made this change too. Now the labels are :

"traefik.http.middlewares.monica-stripprefix.stripprefix.prefixes": "`/monica`",
"traefik.http.routers.monica.middlewares": "monica-stripprefix@docker",
"traefik.http.routers.monica.rule": "Host(`rpihome`) && Path(`/monica`)"

Same behaviour like before. The container is reached with a GET on /monica that get 404ed.

Here are some screenshots from my traefik dashboard, maybe that helps:

Still very very confused with this behaviour, and still trying to wrap my head around this. I can also try and share my terraform files if anyone wants to replicate this behaviour locally!

Thanks.

[bluepuma77]

Not sure what Terraform is doing, but in general stripprefix works.

    labels:
      - traefik.http.middlewares.whoami-strip.stripprefix.prefixes=/test
      - traefik.http.routers.whoami.middlewares=whoami-strip

https://whoami.example.com/test -> /
https://whoami.example.com/blub -> /blub

Example docker-compose.yml, it works with and without @docker in middleware assignment to router

version: '3.9'

services:
  traefik:
    image: traefik:v2.9
    ports:
      # listen on host ports without ingress network (Docker Swarm)
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
    networks:
      - proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - traefik-certificates:/certificates
    command:
      --providers.docker=true
      --providers.docker.exposedByDefault=false
      --entryPoints.web.address=:80
      --entryPoints.web.http.redirections.entryPoint.to=websecure
      --entryPoints.web.http.redirections.entryPoint.scheme=https
      --entryPoints.websecure.address=:443
      --entryPoints.websecure.http.tls=true
      --api.debug=true
      --api.dashboard=true
      --log.level=DEBUG
      --accesslog=true
      --certificatesResolvers.myresolver.acme.email=EMAIL@example.com
      --certificatesResolvers.myresolver.acme.storage=/certificates/acme.json
      --certificatesResolvers.myresolver.acme.httpchallenge.entrypoint=web
    labels:
      - traefik.enable=true
      - traefik.http.routers.api.entrypoints=websecure
      - traefik.http.routers.api.rule=Host(`traefik.example.com`)
      - traefik.http.routers.api.tls.certresolver=myresolver
      - traefik.http.routers.api.service=api@internal
      - traefik.http.routers.api.middlewares=auth
      - 'traefik.http.middlewares.auth.basicauth.users=test:$$apr1$$H6uskkkW$$IgXLP6ewTrSuBkTrqE8wj/'

  whoami:
    image: traefik/whoami:v1.8
    networks:
      - proxy
    labels:
      - traefik.enable=true
      - traefik.http.routers.whoami.entrypoints=websecure
      - traefik.http.routers.whoami.rule=Host(`whoami.example.com`)
      - traefik.http.routers.whoami.tls.certresolver=myresolver
      - traefik.http.services.whoami.loadbalancer.server.port=80
      - traefik.http.middlewares.whoami-strip.stripprefix.prefixes=/test
      - traefik.http.routers.whoami.middlewares=whoami-strip

networks:
  proxy:
    name: proxy
    external: true

volumes:
  traefik-certificates:

[level0-coder]

Thank you once again for you answer. I wanted to make sure that the terraform provider for docker isn't doing anything different while creating labels for the container, so I created a simple docker-compose file and I'm still not able to make it work.

In your example, I can see that https://whoami.example.com/test -> / works very well. However I would be more interested to see if https://example.com/whoami -> / will work by changing the labels as follows for the whoami service:

traefik.http.routers.whoami.rule=Host(`whoami.example.com`)

to

traefik.http.routers.whoami.rule=Path(`/whoami`)

and

traefik.http.middlewares.whoami-strip.stripprefix.prefixes=/test

to

traefik.http.middlewares.whoami-strip.stripprefix.prefixes=/whoami

This would be exactly what I am trying to do.

[bluepuma77]

It should work the way you put it. The only potential problem I can see is that TLS with Lets Encrypt will probably not work if you don't have a Host.

When you combine Host with Path for the rule it should be fine. Example:

(Host(`example.org`) && Path(`/traefik`))

[level0-coder]

@bluepuma77

When you combine Host with Path for the rule it should be fine.

Still no success, and weirdly in this case the GET request /monica is not even forwarded to the service, but rather I'm getting 302ed to /register.

Here is my docker-compose.yaml for reference:

version: "3.3"

services:
  traefik:
    image: traefik:v2.9
    ports:
      - 80:80
      - 8080:8080
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command:
      - --api.insecure=true
      - --providers.docker
      - --accesslog=true
      - --log.level=DEBUG
      - --log.format=json

  mysql:
    image: servercontainers/mysql
    environment:
      - MYSQL_RANDOM_ROOT_PASSWORD=true
      - ADMIN_USER=someuser
      - ADMIN_PASSWORD=somerandompassword
      - DB_NAME=monica
      - DB_USER=homestead
      - DB_PASSWORD=secret
      - BACKUP_ENABLED=false
    volumes:
      - /tmp/mysql-data:/var/lib/mysql
      - /tmp/mysqlbackup:/var/mysql-backup

  monica:
    image: monica:latest
    environment:
      - DB_HOST=mysql
    labels:
      - traefik.enable=true
      - traefik.http.routers.monica.rule=(Host(`rpihome`) && Path(`/monica`))
      - traefik.http.middlewares.monica-stripprefix.stripprefix.prefixes=/monica
      - traefik.http.routers.monica.middlewares=monica-stripprefix
      - traefik.http.services.monica.loadbalancer.server.port=80

volumes:
  monica_volume:
    external: true

[bluepuma77]

That 302 to /register indicates to me that it's working. You are not logged in, so monica sends you to the /register page, which probably enables login, too.

The issue is that monica does not know about the prefix, so it will redirect you to /register and not to /monica/register.

There are other applications that will not work with a custom prefix. My recommendation: use a sub-domain, will make life easier.

[level0-coder]

Wow, that makes so much sense, thanks a ton.

I would love to use a subdomain, however at the moment I am keeping things TLS free, so I can understand certs and HTTPS better once I grab the concept of routing and networking properly.

So at the moment my Host is rpihome and my network has been configured to use rpihome as a dns for my raspberry pi ( I do this using tailscale ). Somehow at the moment I am not able to get subdomain work with this setup.

Do you have any advice for me in this situation, I would still like to work without TLS for the moment.

[bluepuma77]

Check if you can configure your router to add a custom DNS entry to point monica to the same IP as rpihome.

If you only use a single PC/Mac to access monica, you can just add it to your hosts file.

[level0-coder]

Sadly I have multiple devices on my network and my VPN makes it easier to connect to my devices from anywhere in the world. I will see if there is a way to use subdomains on my vpn, if not I will use the whoami example to first learn concepts of TLS and then have everything run on my domain, so subdomains will be easier.

@bluepuma77 Thanks a lot for you help on this one, I learned a lot with your help.

Greetings.

[level0-coder]

Update: It was actually super easy to get this working.

I already had a domain lying around, and all I had to do was to create an A record that pointed to the tailnet IP address of my raspberry pi. Once I did that monica.domain.com was redirected to my monica instance.

Thanks again for all the help.