Middleware ipAllowList SOMETIMES returns 403 Forbidden on local network

Hi, moved to Traefik quite recently after using NPM for a long time - so far I'm loving Traefik very much.

But I got a strange issue
so my setup is as following (as a preface - I'm using traefik not in Docker, and non of my services are in docker, they're all on VMs/LXCs with their own IPs)

  1. I have a home lab with a local DNS and broadcasting it via my router.
  2. I got a public domain going to my home lab - XYZ.com and port forwarded from my router to my Traefik VM (previously my NPM VM)
  3. I had (on NPM) a local domain - local.XYZ.com that suppose to serve services ONLY on local network
  4. All public services were accessibly on 1.XYZ.com, 2.XYZ.com and all local services on 3.local.XYZ.com and so on
  5. ALL services (both public and internal) had a valid LE cert and redirected from http -> https

In my current setup -
I've replaced NPM with Traefik
I could not find a way to set up a *.local.XYZ.com setup (if anybody knows, please if you could share?)
In the meantime I've set ALL services on my public domain XYZ.com, and added a ipAllowList middleware only for the local services (figured I'd solve this issue like that)
The issue that happened - is that SOMETIMES (I could not figure a reproducible way to make that happen) even though I'm on my local network - accessing any of the local services, I'd still get a 403 Forbidden error (after several minutes it passes, so if I use a different browser, but not always) - which makes me think that I've setup something wrong and make me nervous as well to host all those on my public domain to start with..

Thank you for reading so far, next i'll post of snippets from my setup to help you see the current place i'm in:

entryPoints:

entryPoints:
  websecure:
    address: :443
    proxyProtocol:
      insecure: false
      trustedIPs: 
        - "127.0.0.1/32"
        - "192.168.1.1/24"
    forwardedHeaders:
      insecure: false
      trustedIPs: 
        - "127.0.0.1/32"
        - "192.168.1.1/24"
    http:
      tls:
        certResolver: cloudflare
        domains:
          - main: "*.XYZ.com"
            sans:
              - "XYZ.com"

middleware:

http:
  middlewares:
    local-iplist:
      ipAllowList:
        sourceRange:
          - "127.0.0.1/32"
          - "192.168.1.1/24"

internal service and router:

services:
    serviceA:
      loadBalancer:
        servers:
          - url: "http://ip:port"
routers:
    routerA:
      rule: "Host(`serviceA.XYZ.com`)"
      entryPoints:
        - websecure
      middlewares:
        - local-iplist
      service: "serviceA"

If anybody needs anymore information, please reply and i'll add it promptly.

Thank you in advance.

Example for a request that resulted in 403:

Request:

authority: serviceA.XYZ.com
:method: GET
:path: /
:scheme: https
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-US,en;q=0.9
Cache-Control: no-cache
Pragma: no-cache
Priority: u=0, i
Sec-Ch-Ua: "Not/A)Brand";v="8", "Chromium";v="126", "Google Chrome";v="126"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "OS"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36

Response:

Alt-Svc: h3=":443"; ma=86400
Cf-Cache-Status: DYNAMIC
Cf-Ray: XXXXXX
Content-Length: 9
Date: Mon, 22 Jul 2024 11:11:20 GMT
Nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=OKk%KK%3D"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Strict-Transport-Security: max-age=15552000; includeSubDomains

Based on this should I expect maybe the issue is with CloudFlare and not my local setup?

Traefik LetsEncrypt wildcard TLS certs can easily be created with main/sans (doc).

entryPoints:
  websecure:
    address: ':443'
    http:
      tls:
        options: foobar
        certResolver: leresolver
        domains:
          - main: example.com
            sans:
              - *.example.com
          - main: local.example.com
            sans:
              - *.local.example.com

But it only works with more complicated dnsChallenge (doc).

What do you do with proxyProtocol? Which other component is using that?

Enable and check Traefik dashboard, Traefik debug log and Traefik access log.

I tried what you suggested, on traefik.log i'm getting this warning

{
  "level": "warn",
  "providerName": "cloudflare.acme",
  "acmeCA": "https://acme-v02.api.letsencrypt.org/directory",
  "ACME CA": "https://acme-v02.api.letsencrypt.org/directory",
  "routerName": "serviceA@file",
  "rule": "Host(`serviceA.XYZ.com`)",
  "time": "2024-07-22T13:45:26+02:00",
  "message": "Domain \"local.XYZ.com\" is duplicated in the configuration or validated by the domain {XYZ.com [*.XYZ.com]}. It will be processed once."
}

So far including *.local.XYZ.com, instead of 403 Forbidden, SOMETIMES i'm getting this SSL_VERSION error

As well - set up a ping collector on my grafana dashboard
The following is a HTTP response from one of the public services

The this is from one of the internal services

It's clear that I've setup something wrong - just not sure what.

Could anybody please assist?

Thank you.

Enable and check Traefik dashboard, Traefik debug log and Traefik access log.

Sorry I forgot to post that, I've had debug status since yesterday...
Whenever the local service is unavailable, there's nothing in traefik.log except for:

{"level":"debug","time":"2024-07-23T08:21:20+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 27e473cf6a79f769"}
{"level":"debug","time":"2024-07-23T08:21:25+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 5adc64e275f8a823"}
{"level":"debug","time":"2024-07-23T08:21:30+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 99e89866d4547548"}
{"level":"debug","time":"2024-07-23T08:21:30+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 27e473cf6a79f769"}
{"level":"debug","time":"2024-07-23T08:21:31+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 5adc64e275f8a823"}
{"level":"debug","time":"2024-07-23T08:21:31+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 5adc64e275f8a823"}
{"level":"debug","time":"2024-07-23T08:21:38+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 5adc64e275f8a823"}
{"level":"debug","time":"2024-07-23T08:21:40+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 99e89866d4547548"}
{"level":"debug","time":"2024-07-23T08:21:40+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 27e473cf6a79f769"}
{"level":"debug","time":"2024-07-23T08:21:42+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 99e89866d4547548"}
{"level":"debug","time":"2024-07-23T08:21:47+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 5adc64e275f8a823"}
{"level":"debug","time":"2024-07-23T08:21:49+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 5adc64e275f8a823"}
{"level":"debug","time":"2024-07-23T08:21:50+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 99e89866d4547548"}
{"level":"debug","time":"2024-07-23T08:21:50+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 5adc64e275f8a823"}
{"level":"debug","time":"2024-07-23T08:21:50+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 27e473cf6a79f769"}
{"level":"debug","time":"2024-07-23T08:21:54+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 5adc64e275f8a823"}
{"level":"debug","time":"2024-07-23T08:22:00+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 99e89866d4547548"}
{"level":"debug","time":"2024-07-23T08:22:00+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 27e473cf6a79f769"}
{"level":"debug","time":"2024-07-23T08:22:01+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 5adc64e275f8a823"}

And in access.log there's nothing, as if it doesn't even reach it.

Share your full Traefik static and dynamic config.

Static:

################################################################
#
# Configuration sample for Traefik v2.
#
# For Traefik v1: https://github.com/traefik/traefik/blob/v1.7/traefik.sample.toml
#
################################################################

################################################################
# Global configuration
################################################################
global:
  checkNewVersion: true
  sendAnonymousUsage: true

################################################################
# EntryPoints configuration
################################################################
entryPoints:
  websecure:
    address: :443
    http:
      tls:
        certResolver: cloudflare
        domains:
          - main: "XYZ.com"
            sans:
              - "*.XYZ.com"
          - main: "local.XYZ.com"
            sans:
              - "*.local.XYZ.com"

################################################################
# Traefik logs configuration
################################################################

log:
  level: DEBUG
  filePath: /path/to/traefik.log
  format: json

################################################################
# Access logs configuration
################################################################

accessLog:
  filePath: /path/to/access.log
  bufferingSize: 100
  fields:
    defaultMode: keep
    names:
      StartUTC: drop
  filters:
    statusCodes:
      - 200
      - 300-302
      - 400-499
    retryAttempts: true
    minDuration: 10ms

################################################################
# API and dashboard configuration
################################################################

api:
  dashboard: true

################################################################
# Provider configuration backend
################################################################

providers:
  file:
    filename: /path/to/configuration.yaml
    watch: true

################################################################
# Cert configuration
################################################################

certificatesResolvers:
  cloudflare:
    acme:
      email: email@email.com
      storage: /path/to/acme.json
      dnsChallenge:
        provider: cloudflare
        resolvers:
          - 1.1.1.1:53
          - 8.8.8.8:53

################################################################
# Metric configuration
################################################################

metrics:
  addInternals: true
  prometheus:
    addRoutersLabels: true
    addEntryPointsLabels: true
    addServicesLabels: true
    entryPoint: metrics
    headerLabels:
      useragent: User-Agent

Dynamic:


################################################################
# HTTP configuration
################################################################


http:

################################################################
# Middlewares configuration
################################################################

  middlewares:
    local-iplist:
      ipAllowList:
        sourceRange:
          - "127.0.0.1/32"
          - "192.168.1.1/24"

    secHeaders:
      headers:
        browserXssFilter: true
        contentTypeNosniff: true
        frameDeny: true
        sslRedirect: true
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 31536000
        customFrameOptionsValue: SAMEORIGIN


################################################################
# Services configuration
################################################################
  services:
    serviceA:
      loadBalancer:
        servers:
          - url: "http://ip:port"
    serviceB:
      loadBalancer:
        servers:
          - url: "http://ip:port"
    serviceC:
      loadBalancer:
        servers:
          - url: "http://ip:port"

################################################################
# Routers configuration
################################################################

  routers:

    dashboard:
      rule: "Host(`dash.local.XTZ.com`)"
      service: api@internal
      middlewares:
        - local-iplist

    routerA:
      rule: "Host(`serviceA.local.XTZ.com`)"
      entryPoints:
        - websecure
      middlewares:
        - local-iplist
      service: "serviceA"
    routerB:
      rule: "Host(`serviceB.local.XTZ.com`)"
      entryPoints:
        - websecure
      middlewares:
        - local-iplist
      service: "serviceB"
    routerC:
      rule: "Host(`serviceC.local.XTZ.com`)"
      entryPoints:
        - websecure
      middlewares:
        - local-iplist
      service: "serviceC"

@bluepuma77 thank you for assisting!

Would love the see the plain difference between your internal and external routers/services, as that seems to be where the problem is.

There isn't much different between them

# External Service
    publicServiceA:
      loadBalancer:
        servers:
          - url: "http://ip:port"
# External Router
    publicRouterA:
      rule: "Host(`publicServiceA.XYZ.com`)"
      entryPoints:
        - websecure
      service: "publicServiceA"

The only visible different is that on the internal routers I have a middleware entry, but the services are identical in definition