Containers getting infinite redirect

Traefik Traefik v2
1

Hello,

I wanted to setup my environment for v3.

I don't use any specifics that were removed in v3, so I changed the compose file to use the v3 image, but when I did so, my containers stopped to be accessible.

I reverted back to 2.6, and they are still inaccessible, and I have no idea why.

The external services defined in yml files work fine, it's just for the container ones that fail.

Example files:

---
version: "3"
services:
  paste-web:
    image: privatebin/nginx-fpm-alpine
    container_name: paste-web
    volumes:
      - /opt/docker/data/paste-data:/srv/data
      - ./conf.php:/srv/cfg/conf.php
    restart: unless-stopped
    networks:
      - traefik
    labels:
      - "traefik.http.routers.paste-web.rule=Host(`paste.example.com`)"
      - "traefik.enable=true"
      - "traefik.docker.network=traefik"
      - "traefik.http.routers.paste-web.entrypoints=web,websecure"
      - "traefik.http.services.paste-web.loadbalancer.server.port=8080"
      - "traefik.http.routers.paste-web.tls=true"
networks:
  traefik:
    external: true

Traefik config:

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    watch: true
    exposedByDefault: false
  file:
    directory: "/etc/traefik/"
    watch: true

entryPoints:
  web:
    address: :80
    http:
      redirections:
        entryPoint:
          to: :443
          scheme: https
          priority: 1
    forwardedHeaders:
      insecure: true
  websecure:
    address: :443

api:
  dashboard: true
  debug: true

serversTransport:
  insecureSkipVerify: true

log:
  level: DEBUG

What happens when I try to connect :

HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://paste.example.com/ [following]
--2023-07-24 11:43:01--  https://paste.example.com/
Reusing existing connection to paste.example.com:443.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://paste.example.com/ [following]
--2023-07-24 11:43:01--  https://paste.example.com/
Reusing existing connection to paste.example.com:443.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://paste.example.com/ [following]
--2023-07-24 11:43:01--  https://paste.example.com/
Reusing existing connection to paste.example.com:443.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://paste.example.com/ [following]
--2023-07-24 11:43:01--  https://paste.example.com/
Reusing existing connection to paste.example.com:443.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://paste.example.com/ [following]
--2023-07-24 11:43:02--  https://paste.example.com/
Reusing existing connection to paste.example.com:443.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://paste.example.com/ [following]
--2023-07-24 11:43:02--  https://paste.example.com/
Reusing existing connection to paste.example.com:443.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://paste.example.com/ [following]
--2023-07-24 11:43:02--  https://paste.example.com/
Reusing existing connection to paste.example.com:443.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://paste.example.com/ [following]
--2023-07-24 11:43:02--  https://paste.example.com/
Reusing existing connection to paste.example.com:443.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://paste.example.com/ [following]
--2023-07-24 11:43:02--  https://paste.example.com/
Reusing existing connection to paste.example.com:443.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://paste.example.com/ [following]
--2023-07-24 11:43:02--  https://paste.example.com/
Reusing existing connection to paste.example.com:443.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://paste.example.com/ [following]
--2023-07-24 11:43:02--  https://paste.example.com/
Reusing existing connection to paste.example.com:443.
HTTP request sent, awaiting response... ^C

If I try the same thing in Traefik v3 I get a http 418 which doesn't help much.

I tried to remove the forwardedHeaders part of the config, it doesn't change anything.

I am really puzzled by this, and I have no idea what's going on.

Thank you for your help!

2

The DEBUG logs of traefik don't include any relevant info.

When I try to connect, it prints something like this for every occurence:

traefik  | time="2023-07-24T09:51:11Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,en;q=0.9\"],\"Cache-Control\":[\"max-age=0\"],\"Sec-Ch-Ua\":[\"\\\"Not/A)Brand\\\";v=\\\"99\\\", \\\"Google Chrome\\\";v=\\\"115\\\", \\\"Chromium\\\";v=\\\"115\\\"\"],\"Sec-Ch-Ua-Mobile\":[\"?0\"],\"Sec-Ch-Ua-Platform\":[\"\\\"Windows\\\"\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Fetch-Site\":[\"none\"],\"Sec-Fetch-User\":[\"?1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36\"],\"X-Forwarded-Host\":[\"paste.example.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"b4883a9ba200\"],\"X-Real-Ip\":[\"10.16.142.225\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"paste.example.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.16.142.225:60455\",\"RequestURI\":\"/\",\"TLS\":null}"
3

docker
docker1439×788 32.9 KB

Here is what the service looks like in the dashboard

4

You don’t show any middlewares, so a redirect can not be created by it. The only redirect is on entrypoint from port 80 to 443.

Check your /etc/traefik/ folder if there are any old config files that are used by Traefik.

5

Exactly my point.

I have no middleware setup in this and yet, I get loop 301.
Traefik Docker compose :

[root@docker-server traefik]# cat /opt/docker/defs/traefik/docker-compose.yml
version: '3'
services:
  traefik:
    image: traefik:v2.6
    container_name: "traefik"
    restart: unless-stopped
    networks:
      - traefik
    # Enables the web UI and tells Traefik to listen to docker
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /opt/docker/data/traefik/config/:/etc/traefik:ro

networks:
  traefik:
    external: true

My config folder

[root@docker-server traefik]# ls -l /opt/docker/data/traefik/config
total 8
drwxr-xr-x. 2 root root 154 Jul 24 13:16 certs
-rw-r--r--. 1 root root 612 Jul 24 13:16 dynamic.yml
-rw-r--r--. 1 root root 437 Jul 24 13:16 traefik.yml

Dynamic file (for certs)

[root@docker-server traefik]# cat /opt/docker/data/traefik/config/dynamic.yml
tls:
  certificates:
    - certFile: "/etc/traefik/certs/example1.com.cert.pem"
      keyFile: "/etc/traefik/certs/example1.com.key.pem"
    - certFile: "/etc/traefik/certs/example2.com.cert.pem"
      keyFile: "/etc/traefik/certs/example2.com.key.pem"
  stores:
    default:
      defaultCertificate:
        certFile: "/etc/traefik/certs/example1.com.cert.pem"
        keyFile: "/etc/traefik/certs/example1.com.key.pem"

Static config:

[root@docker-server traefik]# cat /opt/docker/data/traefik/config/traefik.yml
providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    watch: true
    exposedByDefault: false
  file:
    directory: "/etc/traefik/"
    watch: true

entryPoints:
  web:
    address: :80
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
  websecure:
    address: :443

api:
  dashboard: true
  debug: true

serversTransport:
  insecureSkipVerify: true

log:
  level: DEBUG

Certs folder (to show that it's not holding extra config).

[root@docker-server traefik]# ls -l /opt/docker/data/traefik/config/certs/
total 16
-rw-r--r--. 1 root root 1992 Jul 24 13:16 example1.com.cert.pem
-rw-r--r--. 1 root root 1705 Jul 24 13:16 example1.com.key.pem
-rw-r--r--. 1 root root 2732 Jul 24 13:16 example2.com.cert.pem
-rw-r--r--. 1 root root 3243 Jul 24 13:16 example2.com.key.pem

(and of course I restarted / downed the container multiple times).

[root@docker-server traefik]# docker exec -it traefik ls -l /etc/traefik
total 8
drwxr-xr-x    2 root     root           154 Jul 24 11:16 certs
-rw-r--r--    1 root     root           612 Jul 24 11:16 dynamic.yml
-rw-r--r--    1 root     root           437 Jul 24 11:16 traefik.yml
6

In addition, the container serves perfectly when contacted from the docker network IP :

[root@docker-server traefik]# curl http://172.19.0.4:8080 | head -n 10
<html lang="en">
        <head>
                <meta charset="utf-8" />
                <meta http-equiv="Content-Security-Policy" content="default-src &apos;none&apos;; base-uri &apos;self&apos;; form-action &apos;none&apos;; manifest-src &apos;self&apos;; connect-src * blob:; script-src &apos;self&apos; &apos;unsafe-eval&apos;; style-src &apos;self&apos;; font-src &apos;self&apos;; img-src &apos;self&apos; data: blob:; media-src blob:; object-src blob:">
                <meta http-equiv="X-UA-Compatible" content="IE=edge">
                <meta name="viewport" content="width=device-width, initial-scale=1">
                <meta name="robots" content="noindex" />
                <meta name="google" content="notranslate">
                <title>PrivateBin</title>
7

OK I think I'm making progress.

My environment has a proxy to get out, and it is indeed looping through the web-to-websecure redirect :

traefik  | <PROXY-IP> - - [24/Jul/2023:13:19:10 +0000] "GET / HTTP/1.1" 301 17 "-" "-" 51 "web-to-websecure@internal" "-" 0ms
traefik  | <MY-IP>  - - [24/Jul/2023:13:19:10 +0000] "GET / HTTP/2.0" 301 17 "-" "-" 50 "paste-web@docker" "http://172.19.0.4:8080" 103ms
traefik  | <PROXY-IP> - - [24/Jul/2023:13:19:10 +0000] "GET / HTTP/1.1" 301 17 "-" "-" 53 "web-to-websecure@internal" "-" 0ms
traefik  | <MY-IP>  - - [24/Jul/2023:13:19:10 +0000] "GET / HTTP/2.0" 301 17 "-" "-" 52 "paste-web@docker" "http://172.19.0.4:8080" 100ms
traefik  | <PROXY-IP> - - [24/Jul/2023:13:19:10 +0000] "GET / HTTP/1.1" 301 17 "-" "-" 55 "web-to-websecure@internal" "-" 0ms
traefik  | <MY-IP>  - - [24/Jul/2023:13:19:10 +0000] "GET / HTTP/2.0" 301 17 "-" "-" 54 "paste-web@docker" "http://172.19.0.4:8080" 101ms
traefik  | <PROXY-IP> - - [24/Jul/2023:13:19:10 +0000] "GET / HTTP/1.1" 301 17 "-" "-" 57 "web-to-websecure@internal" "-" 0ms

I see that my docker container has http_proxy and https_proxy, but I don't see how this would play a role in the current situation.

8

OK I figured it out.

My traefik container was getting the http_proxy & https_proxy setup (maybe pushed from docker ?), and was ignoring the no_proxy settings. Therefore when receiving the request, was going to the proxy to request the internal IP.

I forced the http_proxy & https_proxy values to be empty in my compose file, and the services are now accessible again.

9

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.