Ip_whitelist.go:70 > Rejecting IP 172.28.0.1

Traefik Traefik v3 (latest)
,
1

hello world :wink:

i'm using traefik 3.0.0 and already tried both of these middlewares and got same results

traefik | 2024-05-03T02:13:17Z DBG github.com/traefik/traefik/v3/pkg/middlewares/ipallowlist/ip_allowlist.go:57 > Setting up IPAllowLister with sourceRange: [100.0.0.0/8] entryPointName=websecure middlewareName=backend_ipallowlist@docker middlewareType=IPAllowLister routerName=backend@docker
...
traefik | 2024-05-03T02:20:08Z DBG github.com/traefik/traefik/v3/pkg/middlewares/ipwhitelist/ip_whitelist.go:70 > Rejecting IP 172.28.0.1: "172.28.0.1" matched none of the trusted IPs middlewareName=backend_ipwhitelist@docker middlewareType=IPWhiteLister

172.28.0.1 is a gateway IP of a traefik container and NOT a public IP address, yet when I try to hit my website I go over Internet to reach my destination resource... up until last update, the configuration was working without any issue, and now everything broke without even touching anything((

my configuration:

# grep 'traefik\.' docker-compose.override.yaml
      - "traefik.enable=true"
      - "traefik.docker.network=traefik_default"
      - "traefik.http.middlewares.backend_ipwhitelist.ipwhitelist.sourcerange=100.0.0.0/8"
      - "traefik.http.middlewares.backend_stripprefix.stripprefix.prefixes=/backend"
      - "traefik.http.routers.backend.entrypoints=websecure"
      - "traefik.http.routers.backend.middlewares=backend_stripprefix,backend_ipwhitelist"
      - "traefik.http.routers.backend.rule=Host(`X.Y.Z`) && PathPrefix(`/backend`)"
      - "traefik.http.routers.backend.tls.certresolver=myresolver"
      - "traefik.http.services.backend.loadbalancer.server.port=8000"
#

please advise)
thanks!

2

Share your full Traefik static and dynamic config, and docker-compose.yml if used.

4

as i mentioned traefik runs within container, it's being started by docker-compose using following configuration, including traefik config:

# cat docker-compose.yaml 
services:
  traefik:
    image: traefik:latest
# cat docker-compose.override.yaml 
services:
  traefik:
    container_name: traefik
    command:
      - "--api.insecure=true"
      - "--certificatesresolvers.myresolver.tailscale=true"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
      - "--entrypoints.websecure.address=:443"
      - "--log.level=DEBUG"
      - "--ping=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
    healthcheck:
      test: ["CMD-SHELL","traefik healthcheck --ping"]
      interval: 15s
      timeout: 5s
      retries: 2
    image: traefik:latest
    networks:
      - default
    ports:
      - 127.0.0.1:8080:8080
      - 80:80
      - 443:443
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/run/tailscale/tailscaled.sock:/var/run/tailscale/tailscaled.sock
  whoami:
    image: traefik/whoami
    container_name: "simple-service"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`whoami.localhost`)"
      - "traefik.http.routers.whoami.entrypoints=web"
networks:
  default:
#
5

please let me know if you need anything else from me.. maybe this is a bug?

6

What are you trying to do? This should enable all IPs with 100.x.x.x.

7

i'm trying to use above middleware to limit allowed requests on the client IP, however per my original message, I'm no longer able to access that resource (even though I was able to do that in the past) and via debug, i see that Traefik rejects my request because of source IP isn't matching what I specified, however the IP address that's being presented in logs isn't my public IP address, but rather container' gateway IP..

because it was working before and just stopped on it's own without me touching any configs, i think this may be a bug.

what are your thoughts?

8

With sourcerange 100.* it should reject 172.*.

To me it seems there are more changes than only the update.

9

yes, but the issue Traefik isn't seeing my public IP, it rejects container' gateway IP address, which will always be there as next hop network wise..

i'm not aware of any other changes..

  • 100.* that's tailscale range
  • tailscale is installed on the host same as before, perhaps got an update as well
  • traefik runs within container, that runs on host
  • i'm trying to access over tailscale network (hence over 100.*)
  • traefik refuse access and in logs shows up container' gateway IP address

any ideas where to look?)

10 
$ ssh ABC.XYZ.ts.net who
user   pts/0        2024-05-10 21:48 (100.92.134.25)
$ curl -Iv https://ABC.XYZ.ts.net/backend/docs/
* Host ABC.XYZ.ts.net:443 was resolved.
* IPv6: (none)
* IPv4: 100.115.6.126
*   Trying 100.115.6.126:443...
* Connected to ABC.XYZ.ts.net (100.115.6.126) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / X25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=ABC.XYZ.ts.net
*  start date: May  1 00:31:05 2024 GMT
*  expire date: Jul 30 00:31:04 2024 GMT
*  subjectAltName: host "ABC.XYZ.ts.net" matched cert's "ABC.XYZ.ts.net"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://ABC.XYZ.ts.net/backend/docs/
* [HTTP/2] [1] [:method: HEAD]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: ABC.XYZ.ts.net]
* [HTTP/2] [1] [:path: /backend/docs/]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
> HEAD /backend/docs/ HTTP/2
> Host: ABC.XYZ.ts.net
> User-Agent: curl/8.5.0
> Accept: */*
> 
< HTTP/2 403 
HTTP/2 403 
< content-length: 9
content-length: 9
< date: Sat, 11 May 2024 02:07:21 GMT
date: Sat, 11 May 2024 02:07:21 GMT

< 
* Connection #0 to host ABC.XYZ.ts.net left intact
$
11

@bluepuma77 any ideas?)

12

You wrote an update broke it. Can you roll back to see if it still works with an old version?

13

thanks again for all your time, i really appreciate that)

so, i went back, but update came with few other packages as well.. so it's hard for me to say which package caused that.. but before upgrade Traefik worked and I was using 3.0-rc5, and now there is final/stable 3.0.. i tried rc6 and rc5, got same behavior now..

i'm following docs and your suggestions, yet still unable get it to work again(

per Ip_whitelist.go:70 > Rejecting IP 172.28.0.1 - #10 by alexus

you can see im' accessing my resource using tailscale network (100.x.x.x), however Traefik isn't seeing my public IP and using private IP instead and that's what's breaking whitelist(

14

There are multiple IPs in play here:

  1. Device IP
  2. Tailscale IP
  3. Server IP
  4. Docker IP
15

one can clearly see in curl verbose output that request is over Tailscale (100.x.x.x network), next Docker proxies connection into the Traefik' container, however then it appears like Traefik isn't seeing any IP other it's own gateway IP (which always will be the case as a fundamental network part).

16

@bluepuma77 - Do you suspect there might be a software bug in Traefik Tailscale?

17

I have not used tailscale, so I don’t know.

18

bump

# uname -a
Linux X.Y.Z 6.1.0-21-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.90-1 (2024-05-03) x86_64 GNU/Linux
# cat /etc/debian_version
12.5
# docker --version
Docker version 26.1.4, build 5650f9b
# tailscale --version
1.68.1
  tailscale commit: f02ad839770b7d1453e88d64b02162ce81cdd7a8
  other commit: 8f9cb4763e0650eadf4df8131b57733f2ce10d78
  go version: go1.22.4
# docker pull traefik:latest
latest: Pulling from library/traefik
Digest: sha256:ce3ae40b05fd3407b6fe1713e24916a04e81489cf37b9ebb22b7ae98b12064a1
Status: Image is up to date for traefik:latest
docker.io/library/traefik:latest
# docker compose logs --tail 1 traefik
traefik  | 2024-06-21T16:01:44Z DBG github.com/traefik/traefik/v3/pkg/middlewares/ipallowlist/ip_allowlist.go:80 > Rejecting IP 172.28.0.1: "172.28.0.1" matched none of the trusted IPs middlewareName=mongo-express_ipallowlist@docker middlewareType=IPAllowLister
# docker exec -it traefik netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         172.28.0.1      0.0.0.0         UG        0 0          0 eth0
172.28.0.0      0.0.0.0         255.255.0.0     U         0 0          0 eth0
#