# Trying to serve robots.txt for all hosts but still blocked by host ipWhitelist

**URL:** <https://community.traefik.io/t/trying-to-serve-robots-txt-for-all-hosts-but-still-blocked-by-host-ipwhitelist/11582>\
**Category:** Traefik v2\
**Tags:** middleware\
**Created:** [August 27, 2021, 12:37pm UTC](https://community.traefik.io/t/trying-to-serve-robots-txt-for-all-hosts-but-still-blocked-by-host-ipwhitelist/11582 "2021-08-27T12:37:18Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![edasque](https://avatars.discourse-cdn.com/v4/letter/e/e19b73/32.png) [@edasque](https://community.traefik.io/u/edasque)\
**Post date:** [August 27, 2021, 12:37pm UTC](https://community.traefik.io/t/trying-to-serve-robots-txt-for-all-hosts-but-still-blocked-by-host-ipwhitelist/11582/1 "2021-08-27T12:37:18Z")

</div>

All of my hosts use an IPWhitelist that only allows a few IPs in. But some search engine keep trying again and again even if traefik2 responds Forbidden always. I'd like to set up a robots.txt for all (or if I can't, server the robots.txt of each host). This is how it's currently set up:

```auto
  robots:
    image: nginx:alpine
    container_name: robots
    volumes:
      - ./robots.txt:/usr/share/nginx/html/robots.txt:ro
    labels:
      - traefik.enable=true
      - traefik.http.routers.robots.rule=HostRegexp(`{host:.*}`) && Path(`/robots.txt`)
      - traefik.http.routers.robots.entrypoints=https
      - traefik.http.routers.robots.priority=99

```

However, that doesn't seem to work as it still forwards to the service I call, resulting in being forbidden.

What am I missing? I am on 2.5.1 and the robots service & router does show up (though a mention of the priority).

This is what the log shows when I hit the host on /robots.txt:

```auto
	
time="2021-08-27T12:12:54Z" level=debug msg="rejecting request &{Method:GET URL:/robots.txt Proto:HTTP/2.0 ProtoMajor:2 ProtoMinor:0 Header:map[Accept:[*/*] User-Agent:[curl/7.68.0] X-Forwarded-Host:[subdomainredacted.domainredacted.com] X-Forwarded-Port:[443] X-Forwarded-Proto:[https] X-Forwarded-Server:[831fe7e3f88e] X-Real-Ip:[REDACTED]] Body:0xc0013669f0 GetBody:<nil> ContentLength:0 TransferEncoding:[] Close:false Host:subdomainredacted.domainredacted.com Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr:REDACTED:58602 RequestURI:/robots.txt TLS:0xc000644f20 Cancel:<nil> Response:<nil> ctx:0xc0009e7a70}: \"45.79.181.120\" matched none of the trusted IPs" middlewareName=home-ipwhitelist@file middlewareType=IPWhiteLister

```

and beforehand:

```auto
time="2021-08-27T12:11:16Z" level=debug msg="Added outgoing tracing middleware robots-base-watcher" entryPointName=https routerName=robots@docker middlewareName=tracing middlewareType=TracingForwarder
time="2021-08-27T12:11:16Z" level=debug msg="Creating server 0 http://172.18.0.3:80" entryPointName=https routerName=robots@docker serviceName=robots-base-watcher serverName=0
time="2021-08-27T12:11:16Z" level=debug msg="Creating load-balancer" entryPointName=https routerName=robots@docker serviceName=robots-base-watcher
time="2021-08-27T12:11:16Z" level=debug msg="Creating middleware" entryPointName=https routerName=robots@docker serviceName=robots-base-watcher middlewareName=metrics-service middlewareType=Metrics
time="2021-08-27T12:11:16Z" level=debug msg="Creating middleware" entryPointName=https routerName=robots@docker serviceName=robots-base-watcher middlewareName=pipelining middlewareType=Pipelining	
time="2021-08-27T12:11:16Z" level=debug msg="Added outgoing tracing middleware robots-base-watcher" entryPointName=https routerName=robots@docker middlewareName=tracing middlewareType=TracingForwarder
time="2021-08-27T12:11:16Z" level=debug msg="Creating server 0 http://172.18.0.3:80" entryPointName=https routerName=robots@docker serviceName=robots-base-watcher serverName=0
2021-08-27 08:11:16	
time="2021-08-27T12:11:16Z" level=debug msg="Creating load-balancer" entryPointName=https routerName=robots@docker serviceName=robots-base-watcher
time="2021-08-27T12:11:16Z" level=debug msg="Creating middleware" entryPointName=https routerName=robots@docker serviceName=robots-base-watcher middlewareName=metrics-service middlewareType=Metrics
time="2021-08-27T12:11:16Z" level=debug msg="Creating middleware" entryPointName=https routerName=robots@docker serviceName=robots-base-watcher middlewareName=pipelining middlewareType=Pipelining

```

---

<div class="post-metadata">

**Author:** ![grasshide](https://sea2.discourse-cdn.com/flex020/user_avatar/community.traefik.io/grasshide/32/5214_2.png) [@grasshide](https://community.traefik.io/u/grasshide)\
**Post date:** [February 22, 2026, 3:51pm UTC](https://community.traefik.io/t/trying-to-serve-robots-txt-for-all-hosts-but-still-blocked-by-host-ipwhitelist/11582/2 "2026-02-22T15:51:51Z")

</div>

Did you get this to work after all? I’m interested in the solution as well 🙂

---

<div class="post-metadata">

**Author:** ![bluepuma77](https://avatars.discourse-cdn.com/v4/letter/b/a9adbd/32.png) [@bluepuma77](https://community.traefik.io/u/bluepuma77)\
**Post date:** [February 26, 2026, 12:48pm UTC](https://community.traefik.io/t/trying-to-serve-robots-txt-for-all-hosts-but-still-blocked-by-host-ipwhitelist/11582/3 "2026-02-26T12:48:52Z")

</div>

It should work just with `Path()`, make sure to set priotity high enough.

```auto
    labels:
      - traefik.enable=true
      - traefik.http.routers.robots.rule=Path(`/robots.txt`)
      - traefik.http.routers.robots.entrypoints=https
      - traefik.http.routers.robots.priority=9999

```

Enable Traefik access log in JSON format to see the target service used for every request.