Middleware ipAllowList SOMETIMES returns 403 Forbidden on local network

Ocelot · July 22, 2024, 10:30am

Hi, moved to Traefik quite recently after using NPM for a long time - so far I'm loving Traefik very much.

But I got a strange issue
so my setup is as following (as a preface - I'm using traefik not in Docker, and non of my services are in docker, they're all on VMs/LXCs with their own IPs)

I have a home lab with a local DNS and broadcasting it via my router.
I got a public domain going to my home lab - XYZ.com and port forwarded from my router to my Traefik VM (previously my NPM VM)
I had (on NPM) a local domain - local.XYZ.com that suppose to serve services ONLY on local network
All public services were accessibly on 1.XYZ.com, 2.XYZ.com and all local services on 3.local.XYZ.com and so on
ALL services (both public and internal) had a valid LE cert and redirected from http -> https

In my current setup -
I've replaced NPM with Traefik
I could not find a way to set up a *.local.XYZ.com setup (if anybody knows, please if you could share?)
In the meantime I've set ALL services on my public domain XYZ.com, and added a ipAllowList middleware only for the local services (figured I'd solve this issue like that)
The issue that happened - is that SOMETIMES (I could not figure a reproducible way to make that happen) even though I'm on my local network - accessing any of the local services, I'd still get a 403 Forbidden error (after several minutes it passes, so if I use a different browser, but not always) - which makes me think that I've setup something wrong and make me nervous as well to host all those on my public domain to start with..

Thank you for reading so far, next i'll post of snippets from my setup to help you see the current place i'm in:

entryPoints:

entryPoints:
  websecure:
    address: :443
    proxyProtocol:
      insecure: false
      trustedIPs: 
        - "127.0.0.1/32"
        - "192.168.1.1/24"
    forwardedHeaders:
      insecure: false
      trustedIPs: 
        - "127.0.0.1/32"
        - "192.168.1.1/24"
    http:
      tls:
        certResolver: cloudflare
        domains:
          - main: "*.XYZ.com"
            sans:
              - "XYZ.com"

middleware:

http:
  middlewares:
    local-iplist:
      ipAllowList:
        sourceRange:
          - "127.0.0.1/32"
          - "192.168.1.1/24"

internal service and router:

services:
    serviceA:
      loadBalancer:
        servers:
          - url: "http://ip:port"
routers:
    routerA:
      rule: "Host(`serviceA.XYZ.com`)"
      entryPoints:
        - websecure
      middlewares:
        - local-iplist
      service: "serviceA"

If anybody needs anymore information, please reply and i'll add it promptly.

Thank you in advance.

Ocelot · July 22, 2024, 11:19am

Example for a request that resulted in 403:

Request:

authority: serviceA.XYZ.com
:method: GET
:path: /
:scheme: https
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-US,en;q=0.9
Cache-Control: no-cache
Pragma: no-cache
Priority: u=0, i
Sec-Ch-Ua: "Not/A)Brand";v="8", "Chromium";v="126", "Google Chrome";v="126"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "OS"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36

Response:

Alt-Svc: h3=":443"; ma=86400
Cf-Cache-Status: DYNAMIC
Cf-Ray: XXXXXX
Content-Length: 9
Date: Mon, 22 Jul 2024 11:11:20 GMT
Nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=OKk%KK%3D"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Strict-Transport-Security: max-age=15552000; includeSubDomains

Based on this should I expect maybe the issue is with CloudFlare and not my local setup?

bluepuma77 · July 22, 2024, 11:30am

Traefik LetsEncrypt wildcard TLS certs can easily be created with main/sans (doc).

entryPoints:
  websecure:
    address: ':443'
    http:
      tls:
        options: foobar
        certResolver: leresolver
        domains:
          - main: example.com
            sans:
              - *.example.com
          - main: local.example.com
            sans:
              - *.local.example.com

But it only works with more complicated dnsChallenge (doc).

What do you do with proxyProtocol? Which other component is using that?

Enable and check Traefik dashboard, Traefik debug log and Traefik access log.

Ocelot · July 22, 2024, 11:56am

I tried what you suggested, on traefik.log i'm getting this warning

{
  "level": "warn",
  "providerName": "cloudflare.acme",
  "acmeCA": "https://acme-v02.api.letsencrypt.org/directory",
  "ACME CA": "https://acme-v02.api.letsencrypt.org/directory",
  "routerName": "serviceA@file",
  "rule": "Host(`serviceA.XYZ.com`)",
  "time": "2024-07-22T13:45:26+02:00",
  "message": "Domain \"local.XYZ.com\" is duplicated in the configuration or validated by the domain {XYZ.com [*.XYZ.com]}. It will be processed once."
}

Ocelot · July 22, 2024, 12:31pm

So far including *.local.XYZ.com, instead of 403 Forbidden, SOMETIMES i'm getting this SSL_VERSION error

Ocelot · July 22, 2024, 5:07pm

As well - set up a ping collector on my grafana dashboard
The following is a HTTP response from one of the public services

The this is from one of the internal services

It's clear that I've setup something wrong - just not sure what.

Could anybody please assist?

Thank you.

bluepuma77 · July 23, 2024, 5:22am

Enable and check Traefik dashboard, Traefik debug log and Traefik access log.

Ocelot · July 23, 2024, 6:24am

Sorry I forgot to post that, I've had debug status since yesterday...
Whenever the local service is unavailable, there's nothing in traefik.log except for:

{"level":"debug","time":"2024-07-23T08:21:20+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 27e473cf6a79f769"}
{"level":"debug","time":"2024-07-23T08:21:25+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 5adc64e275f8a823"}
{"level":"debug","time":"2024-07-23T08:21:30+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 99e89866d4547548"}
{"level":"debug","time":"2024-07-23T08:21:30+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 27e473cf6a79f769"}
{"level":"debug","time":"2024-07-23T08:21:31+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 5adc64e275f8a823"}
{"level":"debug","time":"2024-07-23T08:21:31+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 5adc64e275f8a823"}
{"level":"debug","time":"2024-07-23T08:21:38+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 5adc64e275f8a823"}
{"level":"debug","time":"2024-07-23T08:21:40+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 99e89866d4547548"}
{"level":"debug","time":"2024-07-23T08:21:40+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 27e473cf6a79f769"}
{"level":"debug","time":"2024-07-23T08:21:42+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 99e89866d4547548"}
{"level":"debug","time":"2024-07-23T08:21:47+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 5adc64e275f8a823"}
{"level":"debug","time":"2024-07-23T08:21:49+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 5adc64e275f8a823"}
{"level":"debug","time":"2024-07-23T08:21:50+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 99e89866d4547548"}
{"level":"debug","time":"2024-07-23T08:21:50+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 5adc64e275f8a823"}
{"level":"debug","time":"2024-07-23T08:21:50+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 27e473cf6a79f769"}
{"level":"debug","time":"2024-07-23T08:21:54+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 5adc64e275f8a823"}
{"level":"debug","time":"2024-07-23T08:22:00+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 99e89866d4547548"}
{"level":"debug","time":"2024-07-23T08:22:00+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 27e473cf6a79f769"}
{"level":"debug","time":"2024-07-23T08:22:01+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 5adc64e275f8a823"}

And in access.log there's nothing, as if it doesn't even reach it.

bluepuma77 · July 23, 2024, 6:42am

Share your full Traefik static and dynamic config.

Ocelot · July 23, 2024, 6:56am

Static:

################################################################
#
# Configuration sample for Traefik v2.
#
# For Traefik v1: https://github.com/traefik/traefik/blob/v1.7/traefik.sample.toml
#
################################################################

################################################################
# Global configuration
################################################################
global:
  checkNewVersion: true
  sendAnonymousUsage: true

################################################################
# EntryPoints configuration
################################################################
entryPoints:
  websecure:
    address: :443
    http:
      tls:
        certResolver: cloudflare
        domains:
          - main: "XYZ.com"
            sans:
              - "*.XYZ.com"
          - main: "local.XYZ.com"
            sans:
              - "*.local.XYZ.com"

################################################################
# Traefik logs configuration
################################################################

log:
  level: DEBUG
  filePath: /path/to/traefik.log
  format: json

################################################################
# Access logs configuration
################################################################

accessLog:
  filePath: /path/to/access.log
  bufferingSize: 100
  fields:
    defaultMode: keep
    names:
      StartUTC: drop
  filters:
    statusCodes:
      - 200
      - 300-302
      - 400-499
    retryAttempts: true
    minDuration: 10ms

################################################################
# API and dashboard configuration
################################################################

api:
  dashboard: true

################################################################
# Provider configuration backend
################################################################

providers:
  file:
    filename: /path/to/configuration.yaml
    watch: true

################################################################
# Cert configuration
################################################################

certificatesResolvers:
  cloudflare:
    acme:
      email: email@email.com
      storage: /path/to/acme.json
      dnsChallenge:
        provider: cloudflare
        resolvers:
          - 1.1.1.1:53
          - 8.8.8.8:53

################################################################
# Metric configuration
################################################################

metrics:
  addInternals: true
  prometheus:
    addRoutersLabels: true
    addEntryPointsLabels: true
    addServicesLabels: true
    entryPoint: metrics
    headerLabels:
      useragent: User-Agent

Ocelot · July 23, 2024, 6:57am

Dynamic:


################################################################
# HTTP configuration
################################################################


http:

################################################################
# Middlewares configuration
################################################################

  middlewares:
    local-iplist:
      ipAllowList:
        sourceRange:
          - "127.0.0.1/32"
          - "192.168.1.1/24"

    secHeaders:
      headers:
        browserXssFilter: true
        contentTypeNosniff: true
        frameDeny: true
        sslRedirect: true
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 31536000
        customFrameOptionsValue: SAMEORIGIN


################################################################
# Services configuration
################################################################
  services:
    serviceA:
      loadBalancer:
        servers:
          - url: "http://ip:port"
    serviceB:
      loadBalancer:
        servers:
          - url: "http://ip:port"
    serviceC:
      loadBalancer:
        servers:
          - url: "http://ip:port"

################################################################
# Routers configuration
################################################################

  routers:

    dashboard:
      rule: "Host(`dash.local.XTZ.com`)"
      service: api@internal
      middlewares:
        - local-iplist

    routerA:
      rule: "Host(`serviceA.local.XTZ.com`)"
      entryPoints:
        - websecure
      middlewares:
        - local-iplist
      service: "serviceA"
    routerB:
      rule: "Host(`serviceB.local.XTZ.com`)"
      entryPoints:
        - websecure
      middlewares:
        - local-iplist
      service: "serviceB"
    routerC:
      rule: "Host(`serviceC.local.XTZ.com`)"
      entryPoints:
        - websecure
      middlewares:
        - local-iplist
      service: "serviceC"

Ocelot · July 23, 2024, 6:59am

@bluepuma77 thank you for assisting!

bluepuma77 · July 23, 2024, 7:41am

Would love the see the plain difference between your internal and external routers/services, as that seems to be where the problem is.

Ocelot · July 23, 2024, 7:45am

There isn't much different between them

# External Service
    publicServiceA:
      loadBalancer:
        servers:
          - url: "http://ip:port"

# External Router
    publicRouterA:
      rule: "Host(`publicServiceA.XYZ.com`)"
      entryPoints:
        - websecure
      service: "publicServiceA"

The only visible different is that on the internal routers I have a middleware entry, but the services are identical in definition

Topic		Replies	Views
ipAllowList doesn't allowing an allowed IP Traefik v3 (latest) middleware	3	189	October 28, 2024
ipAllowList with httproute does not allow anything Traefik v3 (latest) kubernetes-crd	1	63	November 26, 2024
Odd Traefik Behavior Traefik v3 (latest) docker , middleware	3	89	July 25, 2024
Middleware ipWhitelist not working properly? Traefik v2 middleware	5	2894	August 7, 2023
Assistance with setting up ipAllowList Traefik v3 (latest) docker	1	88	February 11, 2025

Middleware ipAllowList SOMETIMES returns 403 Forbidden on local network

Related topics