Traefik suddenly stopped working with 403 forbidden

Traefik Traefik v3 (latest)
1

Today my traefik suddenly stopped working and reports 403 forbidden for all services. Maybe somebody can give me a hint what and why this happened as I have no clue where to look at.

traefik.conf


api:
  dashboard: true

metrics:
  prometheus:
    addrouterslabels: true

certificatesResolvers:
  http_resolver:
    acme:
      email: "xxx"
      storage: "acme_letsencrypt.json"
      httpChallenge:
        entryPoint: web
  tls_resolver:
    acme:
      tlsChallenge: true
      email: "xxx"
      storage: "tls_letsencrypt.json"
entryPoints:
  ping:
    address: ":88"
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: "websecure"
          scheme: "https"
      middlewares:
        - traefik-crowdsec-bouncer@file
  websecure:
    address: ":443"
    http:
      middlewares:
        - traefik-crowdsec-bouncer@file
    proxyProtocol:
      trustedIPs:
       - 10.0.0.0/8
       - 172.16.0.0/12
       - 192.168.0.0/16
       - 0.0.0.0/0

    forwardedHeaders:
      trustedIPs:
       - 10.0.0.0/8
       - 172.16.0.0/12
       - 192.168.0.0/16
       - 0.0.0.0/0

ping:
  entryPoint: "ping"

global:
  checknewversion: true
  sendanonymoususage: false

experimental:
  plugins:
    real-ip:
      moduleName: github.com/Paxxs/traefik-get-real-ip
      version: "v1.0.2"

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    network: "proxy"
  file:
    filename: "./dynamic_conf.yml"
    watch: true
  providersThrottleDuration: 10

log:
  level: "INFO"
  filePath: "/var/log/traefik/traefik.log"
accessLog:
  filePath: "/var/log/traefik/access.log"
  bufferingSize: 100

access.log:

127.0.0.1 - - [17/Dec/2024:15:38:06 +0000] "HEAD /ping HTTP/1.1" 200 2 "-" "-" 1 "ping@internal" "-" 0ms
127.0.0.1 - - [17/Dec/2024:15:38:16 +0000] "HEAD /ping HTTP/1.1" 200 2 "-" "-" 4 "ping@internal" "-" 0ms
94.31.117.104 - - [17/Dec/2024:15:38:16 +0000] "GET / HTTP/2.0" 403 9 "-" "-" 3 "traefik@docker" "-" 243ms
94.31.117.104 - - [17/Dec/2024:15:38:16 +0000] "GET / HTTP/2.0" 403 9 "-" "-" 2 "traefik@docker" "-" 476ms
94.31.117.104 - - [17/Dec/2024:15:38:18 +0000] "GET / HTTP/1.1" 403 9 "-" "-" 5 "traefik@docker" "-" 507ms
94.31.117.104 - - [17/Dec/2024:15:38:18 +0000] "GET / HTTP/1.1" 403 9 "-" "-" 6 "router-portainer@file" "-" 492ms
94.31.117.104 - - [17/Dec/2024:15:38:19 +0000] "GET / HTTP/2.0" 403 9 "-" "-" 7 "traefik@docker" "-" 13ms
94.31.117.104 - - [17/Dec/2024:15:38:19 +0000] "GET / HTTP/2.0" 403 9 "-" "-" 8 "traefik@docker" "-" 13ms
94.31.117.104 - - [17/Dec/2024:15:38:19 +0000] "GET / HTTP/2.0" 403 9 "-" "-" 9 "traefik@docker" "-" 14ms
94.31.117.104 - - [17/Dec/2024:15:38:20 +0000] "GET / HTTP/2.0" 403 9 "-" "-" 10 "traefik@docker" "-" 13ms
94.31.117.104 - - [17/Dec/2024:15:38:20 +0000] "GET / HTTP/2.0" 403 9 "-" "-" 11 "traefik@docker" "-" 14ms
94.31.117.104 - - [17/Dec/2024:15:38:20 +0000] "GET / HTTP/2.0" 403 9 "-" "-" 12 "traefik@docker" "-" 22ms
94.31.117.104 - - [17/Dec/2024:15:38:20 +0000] "GET / HTTP/2.0" 403 9 "-" "-" 13 "traefik@docker" "-" 20ms
94.31.117.104 - - [17/Dec/2024:15:38:20 +0000] "GET / HTTP/2.0" 403 9 "-" "-" 14 "traefik@docker" "-" 15ms
94.31.117.104 - - [17/Dec/2024:15:38:20 +0000] "GET / HTTP/1.1" 403 9 "-" "-" 15 "router-paperless@file" "-" 21ms
94.31.117.104 - - [17/Dec/2024:15:38:21 +0000] "GET / HTTP/2.0" 403 9 "-" "-" 16 "traefik@docker" "-" 20ms
94.31.117.104 - - [17/Dec/2024:15:38:21 +0000] "GET / HTTP/2.0" 403 9 "-" "-" 17 "traefik@docker" "-" 13ms
94.31.117.104 - - [17/Dec/2024:15:38:22 +0000] "GET /fhem?fw_id= HTTP/2.0" 403 9 "-" "-" 18 "router-fhem@file" "-" 26ms
94.31.117.104 - - [17/Dec/2024:15:38:22 +0000] "GET /favicon.ico HTTP/2.0" 403 9 "-" "-" 19 "router-fhem@file" "-" 15ms
94.31.117.104 - - [17/Dec/2024:15:38:23 +0000] "GET /fhem?fw_id= HTTP/2.0" 403 9 "-" "-" 20 "router-fhem@file" "-" 12ms
94.31.117.104 - - [17/Dec/2024:15:38:23 +0000] "GET /favicon.ico HTTP/2.0" 403 9 "-" "-" 21 "router-fhem@file" "-" 13ms
2

Enable and check Traefik debug log and Traefik access log in JSON format.

I would expect it’s the crowdsec plugin blocking request.

3

Same here. Traefik working for over 1 year and running into issues starting December 17th. I’m running Ubuntu 22.04 and I have a few external web services. When I delete the traefik pod it works for some time, like 5 to 20 seconds. When I do a complete reinstall of microk8s and the entire environment, it might work for longer. I had it working from December 19th to December 21, a little over 30hrs. I’m looking for alternative solutions atm. Just can’t figure it out

4

What exactly is your issue, what error do you see?

What plugins do you use?

Did you upgrade Traefik at that time, did you try to downgrade (releases)?

5

The issue is that my web services 404. I didn’t upgrade, downgrade or change anything. The services just randomly stopped working. The logs do not show the get request and there are no errors in the log. If I delete the pod traefik pod, it works for a few seconds. If remove microk8s and reinstall my entire configuration, last time it worked for over 24hrs. I had traefik working for over a year and then it randomly stopped working. I’m not sure how to diagnose further and I am attempting to use haproxy as we speak

6

Sorry for not responding but I had to get this thing up and running within time. My whole home automation and various other services are running via reverse proxy. Having this not working is really causing trouble. A friend of mine therefore helped me to setup Nginx Proxy Manager and I am really sorry to say - but this one is so much easier to handle. We got it set up in <10 minutes and adding a service takes <30s without digging trough and editing any config file.
So I do not know why my traefic stopped working and I will also not investigate further. The only thing I know for sure is that nobody changed anything, no one was connected to the server and still it stopped working out of nothing.

7

Let me asure you one thing: I support here voluntarily for years and everyone claims that nothing changed, but later 95% find out something did indeed change :wink:

Traefik is not a hobby tool, it is more complex, because it can do a lot of things.

For hobby or home use, I prefer nginx-proxy with companion, which will handle automatic config via env variables.

1 Like
8

Had the same (traefik & crowdsec). The following helped me:
docker exec crowdsec cscli decisions list ' List active decisions of crowdsec
(if not in docker: [sudo] cscli decisions list)

Then delete the decisions that blocks:
docker exec cscli decisions delete --id xxxxxx
(if not in docker: [sudo] cscli decisions delete --id xxxxxx)
xxxxxx = the id from the decision-list