# Middleware ipAllowList SOMETIMES returns 403 Forbidden on local network

**URL:** <https://community.traefik.io/t/middleware-ipallowlist-sometimes-returns-403-forbidden-on-local-network/23665>\
**Category:** Traefik v3 (latest)\
**Tags:** file, middleware\
**Created:** [July 22, 2024, 10:30am UTC](https://community.traefik.io/t/middleware-ipallowlist-sometimes-returns-403-forbidden-on-local-network/23665 "2024-07-22T10:30:18Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ocelot](https://avatars.discourse-cdn.com/v4/letter/o/ec9cab/32.png) [@Ocelot](https://community.traefik.io/u/Ocelot)\
**Post date:** [July 22, 2024, 10:30am UTC](https://community.traefik.io/t/middleware-ipallowlist-sometimes-returns-403-forbidden-on-local-network/23665/1 "2024-07-22T10:30:18Z")

</div>

Hi, moved to Traefik quite recently after using NPM for a long time - so far I'm loving Traefik very much.

But I got a strange issue  
so my setup is as following (as a preface - I'm using traefik not in Docker, and non of my services are in docker, they're all on VMs/LXCs with their own IPs)

1. I have a home lab with a local DNS and broadcasting it via my router.
2. I got a public domain going to my home lab - `XYZ.com` and port forwarded from my router to my Traefik VM (previously my NPM VM)
3. I had (on NPM) a local domain - `local.XYZ.com` that suppose to serve services ONLY on local network
4. All public services were accessibly on `1.XYZ.com, 2.XYZ.com` and all local services on `3.local.XYZ.com` and so on
5. ALL services (both public and internal) had a valid LE cert and redirected from http -\> https

In my current setup -  
I've replaced NPM with Traefik  
I could not find a way to set up a `*.local.XYZ.com` setup (if anybody knows, please if you could share?)  
In the meantime I've set ALL services on my public domain `XYZ.com`, and added a `ipAllowList` middleware only for the local services (figured I'd solve this issue like that)  
The issue that happened - is that SOMETIMES (I could not figure a reproducible way to make that happen) even though I'm on my local network - accessing any of the local services, I'd still get a 403 Forbidden error (after several minutes it passes, so if I use a different browser, but not always) - which makes me think that I've setup something wrong and make me nervous as well to host all those on my public domain to start with..

Thank you for reading so far, next i'll post of snippets from my setup to help you see the current place i'm in:

entryPoints:

```auto
entryPoints:
  websecure:
    address: :443
    proxyProtocol:
      insecure: false
      trustedIPs: 
        - "127.0.0.1/32"
        - "192.168.1.1/24"
    forwardedHeaders:
      insecure: false
      trustedIPs: 
        - "127.0.0.1/32"
        - "192.168.1.1/24"
    http:
      tls:
        certResolver: cloudflare
        domains:
          - main: "*.XYZ.com"
            sans:
              - "XYZ.com"

```

middleware:

```auto
http:
  middlewares:
    local-iplist:
      ipAllowList:
        sourceRange:
          - "127.0.0.1/32"
          - "192.168.1.1/24"

```

internal service and router:

```auto
services:
    serviceA:
      loadBalancer:
        servers:
          - url: "http://ip:port"
routers:
    routerA:
      rule: "Host(`serviceA.XYZ.com`)"
      entryPoints:
        - websecure
      middlewares:
        - local-iplist
      service: "serviceA"

```

If anybody needs anymore information, please reply and i'll add it promptly.

Thank you in advance.

---

<div class="post-metadata">

**Author:** ![Ocelot](https://avatars.discourse-cdn.com/v4/letter/o/ec9cab/32.png) [@Ocelot](https://community.traefik.io/u/Ocelot)\
**Post date:** [July 22, 2024, 11:19am UTC](https://community.traefik.io/t/middleware-ipallowlist-sometimes-returns-403-forbidden-on-local-network/23665/2 "2024-07-22T11:19:03Z")

</div>

Example for a request that resulted in 403:

Request:

```auto
authority: serviceA.XYZ.com
:method: GET
:path: /
:scheme: https
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-US,en;q=0.9
Cache-Control: no-cache
Pragma: no-cache
Priority: u=0, i
Sec-Ch-Ua: "Not/A)Brand";v="8", "Chromium";v="126", "Google Chrome";v="126"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "OS"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36

```

Response:

```auto
Alt-Svc: h3=":443"; ma=86400
Cf-Cache-Status: DYNAMIC
Cf-Ray: XXXXXX
Content-Length: 9
Date: Mon, 22 Jul 2024 11:11:20 GMT
Nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=OKk%KK%3D"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Strict-Transport-Security: max-age=15552000; includeSubDomains

```

Based on this should I expect maybe the issue is with CloudFlare and not my local setup?

---

<div class="post-metadata">

**Author:** ![bluepuma77](https://avatars.discourse-cdn.com/v4/letter/b/a9adbd/32.png) [@bluepuma77](https://community.traefik.io/u/bluepuma77)\
**Post date:** [July 22, 2024, 11:30am UTC](https://community.traefik.io/t/middleware-ipallowlist-sometimes-returns-403-forbidden-on-local-network/23665/3 "2024-07-22T11:30:23Z")

</div>

Traefik LetsEncrypt wildcard TLS certs can easily be created with `main`/`sans` ([doc](https://doc.traefik.io/traefik/routing/entrypoints/#tls)).

```auto
entryPoints:
  websecure:
    address: ':443'
    http:
      tls:
        options: foobar
        certResolver: leresolver
        domains:
          - main: example.com
            sans:
              - *.example.com
          - main: local.example.com
            sans:
              - *.local.example.com

```

But it only works with more complicated `dnsChallenge` ([doc](https://doc.traefik.io/traefik/https/acme/#dnschallenge)).

What do you do with `proxyProtocol`? Which other component is using that?

Enable and check Traefik dashboard, Traefik debug log and Traefik access log.

---

<div class="post-metadata">

**Author:** ![Ocelot](https://avatars.discourse-cdn.com/v4/letter/o/ec9cab/32.png) [@Ocelot](https://community.traefik.io/u/Ocelot)\
**Post date:** [July 22, 2024, 11:56am UTC](https://community.traefik.io/t/middleware-ipallowlist-sometimes-returns-403-forbidden-on-local-network/23665/4 "2024-07-22T11:56:06Z")

</div>

I tried what you suggested, on traefik.log i'm getting this warning

```auto
{
  "level": "warn",
  "providerName": "cloudflare.acme",
  "acmeCA": "https://acme-v02.api.letsencrypt.org/directory",
  "ACME CA": "https://acme-v02.api.letsencrypt.org/directory",
  "routerName": "serviceA@file",
  "rule": "Host(`serviceA.XYZ.com`)",
  "time": "2024-07-22T13:45:26+02:00",
  "message": "Domain \"local.XYZ.com\" is duplicated in the configuration or validated by the domain {XYZ.com [*.XYZ.com]}. It will be processed once."
}

```

---

<div class="post-metadata">

**Author:** ![Ocelot](https://avatars.discourse-cdn.com/v4/letter/o/ec9cab/32.png) [@Ocelot](https://community.traefik.io/u/Ocelot)\
**Post date:** [July 22, 2024, 12:31pm UTC](https://community.traefik.io/t/middleware-ipallowlist-sometimes-returns-403-forbidden-on-local-network/23665/5 "2024-07-22T12:31:26Z")

</div>

So far including `*.local.XYZ.com`, instead of 403 Forbidden, SOMETIMES i'm getting this SSL\_VERSION error

 ![image](https://us1.discourse-cdn.com/flex020/uploads/containo/original/2X/a/a20544c0ab17cdda266d9fc5e592612502b6b096.png)

---

<div class="post-metadata">

**Author:** ![Ocelot](https://avatars.discourse-cdn.com/v4/letter/o/ec9cab/32.png) [@Ocelot](https://community.traefik.io/u/Ocelot)\
**Post date:** [July 22, 2024, 5:07pm UTC](https://community.traefik.io/t/middleware-ipallowlist-sometimes-returns-403-forbidden-on-local-network/23665/6 "2024-07-22T17:07:16Z")

</div>

As well - set up a ping collector on my grafana dashboard  
The following is a HTTP response from one of the public services

 ![image](https://us1.discourse-cdn.com/flex020/uploads/containo/original/2X/4/475d5839e1064eb613ab34f648467820721dcde6.png)

The this is from one of the internal services

 ![image](https://us1.discourse-cdn.com/flex020/uploads/containo/original/2X/0/0bff88d65710e13f45d50e2d159d096513e5d0d8.png)

It's clear that I've setup something wrong - just not sure what.

Could anybody please assist?

Thank you.

---

<div class="post-metadata">

**Author:** ![bluepuma77](https://avatars.discourse-cdn.com/v4/letter/b/a9adbd/32.png) [@bluepuma77](https://community.traefik.io/u/bluepuma77)\
**Post date:** [July 23, 2024, 5:22am UTC](https://community.traefik.io/t/middleware-ipallowlist-sometimes-returns-403-forbidden-on-local-network/23665/7 "2024-07-23T05:22:23Z")

</div>

> Enable and check Traefik dashboard, Traefik debug log and Traefik access log.

---

<div class="post-metadata">

**Author:** ![Ocelot](https://avatars.discourse-cdn.com/v4/letter/o/ec9cab/32.png) [@Ocelot](https://community.traefik.io/u/Ocelot)\
**Post date:** [July 23, 2024, 6:24am UTC](https://community.traefik.io/t/middleware-ipallowlist-sometimes-returns-403-forbidden-on-local-network/23665/8 "2024-07-23T06:24:36Z")

</div>

Sorry I forgot to post that, I've had debug status since yesterday...  
Whenever the local service is unavailable, there's nothing in `traefik.log` except for:

```auto
{"level":"debug","time":"2024-07-23T08:21:20+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 27e473cf6a79f769"}
{"level":"debug","time":"2024-07-23T08:21:25+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 5adc64e275f8a823"}
{"level":"debug","time":"2024-07-23T08:21:30+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 99e89866d4547548"}
{"level":"debug","time":"2024-07-23T08:21:30+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 27e473cf6a79f769"}
{"level":"debug","time":"2024-07-23T08:21:31+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 5adc64e275f8a823"}
{"level":"debug","time":"2024-07-23T08:21:31+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 5adc64e275f8a823"}
{"level":"debug","time":"2024-07-23T08:21:38+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 5adc64e275f8a823"}
{"level":"debug","time":"2024-07-23T08:21:40+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 99e89866d4547548"}
{"level":"debug","time":"2024-07-23T08:21:40+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 27e473cf6a79f769"}
{"level":"debug","time":"2024-07-23T08:21:42+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 99e89866d4547548"}
{"level":"debug","time":"2024-07-23T08:21:47+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 5adc64e275f8a823"}
{"level":"debug","time":"2024-07-23T08:21:49+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 5adc64e275f8a823"}
{"level":"debug","time":"2024-07-23T08:21:50+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 99e89866d4547548"}
{"level":"debug","time":"2024-07-23T08:21:50+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 5adc64e275f8a823"}
{"level":"debug","time":"2024-07-23T08:21:50+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 27e473cf6a79f769"}
{"level":"debug","time":"2024-07-23T08:21:54+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 5adc64e275f8a823"}
{"level":"debug","time":"2024-07-23T08:22:00+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 99e89866d4547548"}
{"level":"debug","time":"2024-07-23T08:22:00+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 27e473cf6a79f769"}
{"level":"debug","time":"2024-07-23T08:22:01+02:00","caller":"github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196","message":"Service selected by WRR: 5adc64e275f8a823"}

```

And in `access.log` there's nothing, as if it doesn't even reach it.

---

<div class="post-metadata">

**Author:** ![bluepuma77](https://avatars.discourse-cdn.com/v4/letter/b/a9adbd/32.png) [@bluepuma77](https://community.traefik.io/u/bluepuma77)\
**Post date:** [July 23, 2024, 6:42am UTC](https://community.traefik.io/t/middleware-ipallowlist-sometimes-returns-403-forbidden-on-local-network/23665/9 "2024-07-23T06:42:37Z")

</div>

Share your full Traefik static and dynamic config.

---

<div class="post-metadata">

**Author:** ![Ocelot](https://avatars.discourse-cdn.com/v4/letter/o/ec9cab/32.png) [@Ocelot](https://community.traefik.io/u/Ocelot)\
**Post date:** [July 23, 2024, 6:56am UTC](https://community.traefik.io/t/middleware-ipallowlist-sometimes-returns-403-forbidden-on-local-network/23665/10 "2024-07-23T06:56:41Z")

</div>

Static:

```auto
################################################################
#
# Configuration sample for Traefik v2.
#
# For Traefik v1: https://github.com/traefik/traefik/blob/v1.7/traefik.sample.toml
#
################################################################

################################################################
# Global configuration
################################################################
global:
  checkNewVersion: true
  sendAnonymousUsage: true

################################################################
# EntryPoints configuration
################################################################
entryPoints:
  websecure:
    address: :443
    http:
      tls:
        certResolver: cloudflare
        domains:
          - main: "XYZ.com"
            sans:
              - "*.XYZ.com"
          - main: "local.XYZ.com"
            sans:
              - "*.local.XYZ.com"

################################################################
# Traefik logs configuration
################################################################

log:
  level: DEBUG
  filePath: /path/to/traefik.log
  format: json

################################################################
# Access logs configuration
################################################################

accessLog:
  filePath: /path/to/access.log
  bufferingSize: 100
  fields:
    defaultMode: keep
    names:
      StartUTC: drop
  filters:
    statusCodes:
      - 200
      - 300-302
      - 400-499
    retryAttempts: true
    minDuration: 10ms

################################################################
# API and dashboard configuration
################################################################

api:
  dashboard: true

################################################################
# Provider configuration backend
################################################################

providers:
  file:
    filename: /path/to/configuration.yaml
    watch: true

################################################################
# Cert configuration
################################################################

certificatesResolvers:
  cloudflare:
    acme:
      email: email@email.com
      storage: /path/to/acme.json
      dnsChallenge:
        provider: cloudflare
        resolvers:
          - 1.1.1.1:53
          - 8.8.8.8:53

################################################################
# Metric configuration
################################################################

metrics:
  addInternals: true
  prometheus:
    addRoutersLabels: true
    addEntryPointsLabels: true
    addServicesLabels: true
    entryPoint: metrics
    headerLabels:
      useragent: User-Agent

```

---

<div class="post-metadata">

**Author:** ![Ocelot](https://avatars.discourse-cdn.com/v4/letter/o/ec9cab/32.png) [@Ocelot](https://community.traefik.io/u/Ocelot)\
**Post date:** [July 23, 2024, 6:57am UTC](https://community.traefik.io/t/middleware-ipallowlist-sometimes-returns-403-forbidden-on-local-network/23665/11 "2024-07-23T06:57:32Z")

</div>

Dynamic:

```auto

################################################################
# HTTP configuration
################################################################

http:

################################################################
# Middlewares configuration
################################################################

  middlewares:
    local-iplist:
      ipAllowList:
        sourceRange:
          - "127.0.0.1/32"
          - "192.168.1.1/24"

    secHeaders:
      headers:
        browserXssFilter: true
        contentTypeNosniff: true
        frameDeny: true
        sslRedirect: true
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 31536000
        customFrameOptionsValue: SAMEORIGIN

################################################################
# Services configuration
################################################################
  services:
    serviceA:
      loadBalancer:
        servers:
          - url: "http://ip:port"
    serviceB:
      loadBalancer:
        servers:
          - url: "http://ip:port"
    serviceC:
      loadBalancer:
        servers:
          - url: "http://ip:port"

################################################################
# Routers configuration
################################################################

  routers:

    dashboard:
      rule: "Host(`dash.local.XTZ.com`)"
      service: api@internal
      middlewares:
        - local-iplist

    routerA:
      rule: "Host(`serviceA.local.XTZ.com`)"
      entryPoints:
        - websecure
      middlewares:
        - local-iplist
      service: "serviceA"
    routerB:
      rule: "Host(`serviceB.local.XTZ.com`)"
      entryPoints:
        - websecure
      middlewares:
        - local-iplist
      service: "serviceB"
    routerC:
      rule: "Host(`serviceC.local.XTZ.com`)"
      entryPoints:
        - websecure
      middlewares:
        - local-iplist
      service: "serviceC"

```

---

<div class="post-metadata">

**Author:** ![Ocelot](https://avatars.discourse-cdn.com/v4/letter/o/ec9cab/32.png) [@Ocelot](https://community.traefik.io/u/Ocelot)\
**Post date:** [July 23, 2024, 6:59am UTC](https://community.traefik.io/t/middleware-ipallowlist-sometimes-returns-403-forbidden-on-local-network/23665/12 "2024-07-23T06:59:13Z")

</div>

@bluepuma77 thank you for assisting!

---

<div class="post-metadata">

**Author:** ![bluepuma77](https://avatars.discourse-cdn.com/v4/letter/b/a9adbd/32.png) [@bluepuma77](https://community.traefik.io/u/bluepuma77)\
**Post date:** [July 23, 2024, 7:41am UTC](https://community.traefik.io/t/middleware-ipallowlist-sometimes-returns-403-forbidden-on-local-network/23665/13 "2024-07-23T07:41:11Z")

</div>

Would love the see the plain difference between your internal and external routers/services, as that seems to be where the problem is.

---

<div class="post-metadata">

**Author:** ![Ocelot](https://avatars.discourse-cdn.com/v4/letter/o/ec9cab/32.png) [@Ocelot](https://community.traefik.io/u/Ocelot)\
**Post date:** [July 23, 2024, 7:45am UTC](https://community.traefik.io/t/middleware-ipallowlist-sometimes-returns-403-forbidden-on-local-network/23665/14 "2024-07-23T07:45:09Z")

</div>

There isn't much different between them

```auto
# External Service
    publicServiceA:
      loadBalancer:
        servers:
          - url: "http://ip:port"

```

```auto
# External Router
    publicRouterA:
      rule: "Host(`publicServiceA.XYZ.com`)"
      entryPoints:
        - websecure
      service: "publicServiceA"

```

The only visible different is that on the internal routers I have a middleware entry, but the services are identical in definition