Hello @floatingpurr and others.
I am struggling with getting the catch-all router/service working. I have read the above and Customzing a global 404 page in Traefik v.2 | Hi I’m Andrea! and to me it is fine. The issue though is, that once a request for non-existing domain hits Traefik, it doesn't generate 404 or go through error middleware as I would expect, but forwards the request to the error service directly. And only the response from that service generates 403 or 404, hits the middleware, goes back to the error service.
My docker is set-up in the following way (through Ansible):
- name: Traefik Docker Container
docker_container:
name: traefik
image: "{{ traefik_docker_image }}"
pull: true
network_mode: host
volumes:
- "{{ traefik_data_directory }}/traefik.toml:/etc/traefik/traefik.toml:ro"
- "/var/run/docker.sock:/var/run/docker.sock:ro"
ports:
- "{{ traefik_port_http }}:80"
- "{{ traefik_port_https }}:443"
- "{{ traefik_port_ui }}:8080"
labels:
traefik.enable: "true"
traefik.http.routers.traefik-secure.entrypoints: "websecure"
traefik.http.routers.traefik-secure.rule: "Host(`traefik.domain.com`)"
traefik.http.routers.traefik-secure.service: "api@internal"
traefik.http.routers.traefik-secure.middlewares: "traefik-whitelist"
traefik.http.routers.traefik.entrypoints: "web"
traefik.http.routers.traefik.rule: "Host(`traefik.domain.com`)"
traefik.http.routers.traefik.service: "api@internal"
traefik.http.routers.traefik.middlewares: "traefik-whitelist"
traefik.http.services.traefik-secure.loadbalancer.server.port: "{{ traefik_port_ui }}"
traefik.http.services.traefik.loadbalancer.server.port: "{{ traefik_port_ui }}"
traefik.http.middlewares.traefik-whitelist.ipwhitelist.ipstrategy.depth: "1"
traefik.http.middlewares.traefik-whitelist.ipwhitelist.sourcerange: "127.0.0.1/32, 192.168.0.0/16"
restart_policy: unless-stopped
memory: 1g
- name: Nginx for Errors Docker Container
docker_container:
name: nginxErrors
image: nginx:latest
pull: true
network_mode: bridge
volumes:
- "{{ traefik_data_directory }}/error-pages:/usr/share/nginx/error-pages"
- "{{ traefik_data_directory }}/nginx/default.conf:/etc/nginx/conf.d/default.conf"
labels:
traefik.enable: "true"
traefik.http.routers.error-router.rule: "HostRegexp(`{host:.+}`)"
traefik.http.routers.error-router.priority: "1"
traefik.http.routers.error-router.entrypoints: "web,websecure"
traefik.http.routers.error-router.middlewares: "error-pages-middleware"
traefik.http.middlewares.error-pages-middleware.errors.status: "400-599"
traefik.http.middlewares.error-pages-middleware.errors.service: "error-pages-service"
traefik.http.middlewares.error-pages-middleware.errors.query: /{status}.html
traefik.http.services.error-pages-service.loadbalancer.server.port: "80"
restart_policy: unless-stopped
The Nginx under error-pages-service is very simple:
server {
listen 80;
server_name localhost;
error_page 401 /401.html;
error_page 403 /403.html;
error_page 404 /404.html;
error_page 405 /405.html;
error_page 407 /407.html;
error_page 408 /408.html;
error_page 409 /409.html;
error_page 410 /410.html;
error_page 411 /411.html;
error_page 412 /412.html;
error_page 413 /413.html;
error_page 416 /411.html;
error_page 418 /412.html;
error_page 421 /421.html;
error_page 429 /429.html;
error_page 500 /500.html;
error_page 502 /502.html;
error_page 503 /503.html;
error_page 504 /504.html;
error_page 505 /505.html;
location / {
root /usr/share/nginx/error-pages;
#internal;
}
}
And this is traefik debug log:
time="2021-01-07T13:08:17Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.0\",\"ProtoMajor\":1,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,en;q=0.5\"],\"Connection\":[\"close\"],\"Dnt\":[\"1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0\"],\"X-Client-Verify\":[\"NONE\"],\"X-Forwarded-For\":[\"192.168.2.10\"],\"X-Forwarded-Host\":[\"non-existent-domain.domain.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"nas\"],\"X-Real-Ip\":[\"192.168.2.10\"],\"X-Tls-Cipher\":[\"TLS_AES_256_GCM_SHA384\"],\"X-Tls-Client-Intercepted\":[\"Unknown\"],\"X-Tls-Protocol\":[\"TLSv1.3\"],\"X-Tls-Sni-Host\":[\"non-existent-domain.domain.com\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"non-existent-domain.domain.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.2.1:64380\",\"RequestURI\":\"/\",\"TLS\":null}"
time="2021-01-07T13:08:17Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.0\",\"ProtoMajor\":1,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,en;q=0.5\"],\"Connection\":[\"close\"],\"Dnt\":[\"1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0\"],\"X-Client-Verify\":[\"NONE\"],\"X-Forwarded-For\":[\"192.168.2.10\"],\"X-Forwarded-Host\":[\"non-existent-domain.domain.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"nas\"],\"X-Real-Ip\":[\"192.168.2.10\"],\"X-Tls-Cipher\":[\"TLS_AES_256_GCM_SHA384\"],\"X-Tls-Client-Intercepted\":[\"Unknown\"],\"X-Tls-Protocol\":[\"TLSv1.3\"],\"X-Tls-Sni-Host\":[\"non-existent-domain.domain.com\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"non-existent-domain.domain.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.2.1:64380\",\"RequestURI\":\"/\",\"TLS\":null}" ForwardURL="http://172.17.0.43:80"
time="2021-01-07T13:08:17Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.0\",\"ProtoMajor\":1,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,en;q=0.5\"],\"Connection\":[\"close\"],\"Dnt\":[\"1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0\"],\"X-Client-Verify\":[\"NONE\"],\"X-Forwarded-For\":[\"192.168.2.10\"],\"X-Forwarded-Host\":[\"non-existent-domain.domain.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"nas\"],\"X-Real-Ip\":[\"192.168.2.10\"],\"X-Tls-Cipher\":[\"TLS_AES_256_GCM_SHA384\"],\"X-Tls-Client-Intercepted\":[\"Unknown\"],\"X-Tls-Protocol\":[\"TLSv1.3\"],\"X-Tls-Sni-Host\":[\"non-existent-domain.domain.com\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"non-existent-domain.domain.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.2.1:64380\",\"RequestURI\":\"/\",\"TLS\":null}"
time="2021-01-07T13:08:17Z" level=error msg="Caught HTTP Status Code 403, returning error page" middlewareName=error-pages-middleware@docker middlewareType=customError
time="2021-01-07T13:08:17Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"0.0.0.0\",\"Path\":\"/403.html\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,en;q=0.5\"],\"Connection\":[\"close\"],\"Dnt\":[\"1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0\"],\"X-Client-Verify\":[\"NONE\"],\"X-Forwarded-For\":[\"192.168.2.10\"],\"X-Forwarded-Host\":[\"non-existent-domain.domain.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"nas\"],\"X-Real-Ip\":[\"192.168.2.10\"],\"X-Tls-Cipher\":[\"TLS_AES_256_GCM_SHA384\"],\"X-Tls-Client-Intercepted\":[\"Unknown\"],\"X-Tls-Protocol\":[\"TLSv1.3\"],\"X-Tls-Sni-Host\":[\"non-existent-domain.domain.com\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"0.0.0.0\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"\",\"RequestURI\":\"/403.html\",\"TLS\":null}"
time="2021-01-07T13:08:17Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"0.0.0.0\",\"Path\":\"/403.html\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,en;q=0.5\"],\"Connection\":[\"close\"],\"Dnt\":[\"1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0\"],\"X-Client-Verify\":[\"NONE\"],\"X-Forwarded-For\":[\"192.168.2.10\"],\"X-Forwarded-Host\":[\"non-existent-domain.domain.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"nas\"],\"X-Real-Ip\":[\"192.168.2.10\"],\"X-Tls-Cipher\":[\"TLS_AES_256_GCM_SHA384\"],\"X-Tls-Client-Intercepted\":[\"Unknown\"],\"X-Tls-Protocol\":[\"TLSv1.3\"],\"X-Tls-Sni-Host\":[\"non-existent-domain.domain.com\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"0.0.0.0\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"\",\"RequestURI\":\"/403.html\",\"TLS\":null}" ForwardURL="http://172.17.0.43:80"
time="2021-01-07T13:08:17Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"0.0.0.0\",\"Path\":\"/403.html\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,en;q=0.5\"],\"Connection\":[\"close\"],\"Dnt\":[\"1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0\"],\"X-Client-Verify\":[\"NONE\"],\"X-Forwarded-For\":[\"192.168.2.10\"],\"X-Forwarded-Host\":[\"non-existent-domain.domain.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"nas\"],\"X-Real-Ip\":[\"192.168.2.10\"],\"X-Tls-Cipher\":[\"TLS_AES_256_GCM_SHA384\"],\"X-Tls-Client-Intercepted\":[\"Unknown\"],\"X-Tls-Protocol\":[\"TLSv1.3\"],\"X-Tls-Sni-Host\":[\"non-existent-domain.domain.com\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"0.0.0.0\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"\",\"RequestURI\":\"/403.html\",\"TLS\":null}"
I removed the 'internal' directive from Nginx as with it I was getting 404 all the time instead of actual error page - something to check independently.
With the above the request hits Nginx and it responds:
172.17.0.1 - - [07/Jan/2021:13:08:17 +0000] "GET / HTTP/1.1" 403 1459 "-" "Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0" "192.168.2.10, 192.168.2.1"
172.17.0.1 - - [07/Jan/2021:13:08:17 +0000] "GET /403.html HTTP/1.1" 200 1459 "-" "Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0" "192.168.2.10"
2021/01/07 13:08:17 [error] 28#28: *8 directory index of "/usr/share/nginx/error-pages/" is forbidden, client: 172.17.0.1, server: localhost, request: "GET / HTTP/1.1", host: "non-existent-domain.domain.com"
So what am I missing?
I have an app that is working fine - the request first hits the ipwhitelist, traefik generates 403 and that 403 is then send to Nginx error service.