Global 404 handler

[gingerbreadassassin]

Is there some way to globally apply default middlewares, e.g. Errors?

Currently, if I specify a subdomain to my instance of traefik 2.0, it serves up the traefik default cert and I don't even get a 404 in my browser, just a warning about my connection not being private.

Here's my stack yml, with some commented-out lines chronicling my struggles with the switch to 2.0:

version: "3.7"

services:
  traefik:
    image: traefik
    command:
      - "--log.level=DEBUG"
      - "--api=true"
      - "--api.dashboard=true"
      - "--providers.docker=true"
      - "--providers.docker.endpoint=unix:///var/run/docker.sock"
      - "--providers.docker.swarmMode=true"
      - "--providers.docker.exposedByDefault=false"
      - "--entryPoints.web.address=:80"
      - "--entryPoints.websecure.address=:443"
      - "--certificatesresolvers.le.acme.email=me@mydomain.example"
      - "--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json"
      - "--certificatesresolvers.le.acme.httpchallenge=true"
      - "--certificatesresolvers.le.acme.httpchallenge.entryPoint=web"
    #   - "--certificatesresolvers.le.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - acme:/letsencrypt
    networks:
      - webgateway
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
    deploy:
      mode: global
      placement:
        constraints:
          - node.role == manager
      restart_policy:
        condition: any
      labels:
        - traefik.enable=true
        - traefik.docker.network=webgateway
        # - traefik.http.routers.web.rule=Host(`*.mydomain.example`)
        # - traefik.http.routers.web.entrypoints=web
        # - traefik.http.routers.web.middlewares=redirect@file
        - traefik.http.routers.catchall.rule=HostRegexp(`mydomain.example`, `{subdomain:[a-z]+}.mydomain.example`)
        - traefik.http.routers.catchall.entrypoints=websecure
        - traefik.http.routers.catchall.tls=true
        - traefik.http.routers.catchall.tls.certresolver=le
        - traefik.http.routers.catchall.middlewares=error404
        # - traefik.http.routers.web.rule=Host(`traefik.mydomain.example`)
        # - traefik.http.routers.web.entrypoints=web
        # - traefik.http.routers.web.middlewares=redirect@file
        - traefik.http.routers.api.rule=Host(`traefik.mydomain.example`)
        - traefik.http.routers.api.entrypoints=websecure
        - traefik.http.routers.api.tls=true
        - traefik.http.routers.api.tls.certresolver=le
        # - traefik.http.routers.websecured.service=traefik
        # - traefik.http.routers.api.rule=PathPrefix(`/api`) || PathPrefix(`/dashboard`)
        - traefik.http.routers.api.service=api@internal
        - traefik.http.routers.api.middlewares=error404,traefik-filter,traefik-auth
        - traefik.http.services.api@internal.loadbalancer.server.port=8080
        # - traefik.http.services.traefik.loadbalancer.passhostheader=true
        - traefik.http.middlewares.traefik-filter.ipwhitelist.sourcerange=123.123.123.123/24
        - traefik.http.middlewares.traefik-auth.basicauth.users=me:secret
        - traefik.http.middlewares.error404.errors.status=404
        - traefik.http.middlewares.error404.errors.service=404
        
  whoami:
    image: containous/whoami
    networks:
        - webgateway
    deploy:
        labels:
          - traefik.enable=true
          - traefik.docker.network=webgateway
          - traefik.http.routers.whoami.rule=Host(`whoami.mydomain.example`)
          - traefik.http.routers.whoami.entrypoints=websecure
          - traefik.http.routers.whoami.tls=true
          - traefik.http.routers.whoami.tls.certresolver=le
          - traefik.http.routers.whoami.service=whoami
          - traefik.http.services.whoami.loadbalancer.server.port=80

volumes:
  acme:
networks:
  webgateway:
    external: true

and the yml for my error handler:

version: "3.7"

services:
  app:
    image: $CI_REGISTRY_IMAGE:$CI_BUILD_REF_NAME
    networks:
      - webgateway
    deploy:
      labels:
        - traefik.enable=true
        - traefik.docker.network=webgateway
        - traefik.http.routers.404.rule=Host(`404.mydomain.example`)
        - traefik.http.routers.404.entrypoints=websecure
        - traefik.http.routers.404.tls=true
        - traefik.http.routers.404.tls.certresolver=le
        - traefik.http.routers.404.service=404
        - traefik.http.services.404.loadbalancer.server.port=8000

networks:
  webgateway:
    external: true

Is there some way to define a default template for any service which has the label traefik.enable=true? If I ever decide to change the name of the 404 handler, or its host, it would be a huge pain to go back and check all the previously defined routers to fix them. Using yaml or toml in the traefik service itself is a non-starter for me.

[przemas75]

You have 'connection not secure' if you point your browser to 404.mydomain.example?
Im not sure if i understand what you want to achive

[gingerbreadassassin]

No, if I point my browser to 404.mydomain.example, I get the expected 404 page with its LE certificate.

If I point my browser to does-not-exist.mydomain.example, I get the default traefik cert. If I use curl -k against does-not-exist.mydomain.example, I get the standard traefik 404 page.

I want all 404 errors for *mydomain.example to be redirected to/handled by 404.mydomain.example

[Ch1ch1]

Hi

You have to define a error middleware in you dynamic configuraiton :

  [http.middlewares.errorhandler.errors]
    status = ["400-599"]
    service = "errorhandler-backend@docker"
    query = "/{status}.html"

Define the router and service on you error handler container :

        labels:
            - "traefik.enable=true"
            
            - "traefik.http.routers.errorhandler.rule=Hostr(`error.domain.com`)"
            - "traefik.http.routers.errorhandler.entrypoints=https"
            - "traefik.http.routers.errorhandler.tls.certresolver=resolver_of_your_choice"
                        
            - "traefik.http.services.errorhandler-backend.loadbalancer.server.port=80"

And then tell all your container that you want to go trought the error middleware :

      labels:
            - "traefik.enable=true"
            
            # Router definition
            - "traefik.http.routers.container_lambda.rule=Hostregexp(`bla.domain.com`)"
            - "traefik.http.routers.container_lambda.entrypoints=https"
            - "traefik.http.routers.container_lambda.tls.certresolver=resolver_of_your_choice"
            - "traefik.http.routers.container_lambda.middlewares=errorhandler@file"

            - "traefik.http.services.container_lambda_service.loadbalancer.server.port=8080"

[gingerbreadassassin]

I was able to get your configuration working using docker dynamic configuration. On traefik's stack file, I added:

labels:
  - traefik.http.middlewares.errorhandler.errors.status=400-599
  - traefik.http.middlewares.errorhandler.errors.service=errorhandler-backend
  - traefik.http.middlewares.errorhandler.errors.query=/{status}

Note that I omit the .html part, as my error handler is a flask app, serving up a template at /{status}

I added the following to my error handler:

labels:
  - traefik.enable=true
  - traefik.docker.network=webgateway
  - traefik.http.routers.errorhandler.rule=Host(`error.mydomain.example`)
  - traefik.http.routers.errorhandler.entrypoints=websecure
  - traefik.http.routers.errorhandler.tls=true
  - traefik.http.routers.errorhandler.tls.certresolver=le
  - traefik.http.routers.errorhandler.service=errorhandler-backend
  - traefik.http.services.errorhandler-backend.loadbalancer.server.port=8000

And on an unrelated service, I added:

labels:
  - traefik.http.routers.unrelated-service.middlewares=errorhandler@docker

This works if I point my browser to the url of that unrelated service and add a path that does not exist, e.g.: https://unrelated-service.mydomain.example/this/path/is/not/valid.html

The error handler's page is displayed but the url stays the same. While this works for defined services, I was hoping to have any undefined services serve up the error page. For example, if I point my browser to http(s)://does-not-exist.mydomain.example, this should deliver a 404 to the browser, since no traefik router or service is defined. I say "should" but that's what I'm trying to achieve. I'm not sure if that's possible. I figure it should be, since without any error handler defined, traefik serves up its own 404 page; just a simple 404 page not found in black and white text.

[skarolis]

After using the configuration my traefik container keeps logging:

level=error msg="Caught HTTP Status Code 404, returning error page" middlewareName=error-handler@file middlewareType=customError,

P.S: Error log keeps repeating after the first error page call/trigger

Is this a normal behaviour. Why is this error level?

[skarolis]

In the end seems it's not an issue. I just had some process continuously pinging traefik's endpoint and the catch rule was processing it with 404 response

[ks_sk]

@skarolis

I just had some process continuously pinging traefik's endpoint and the catch rule was processing it with 404 respon

Hi Did you get this working as I have similar requirement to sere a custom error page for any route that is not configured on traefik.

[floatingpurr]

Alright, sorry for pulling up this old post but I'd like to customize the default "404 page not found" message returned by Traefik, when any router rule matches.

I went through this thread and I tried to harness the ErrorPage middleware for that purpose.

Quick recap

It looks like you can use such a middleware just for customizing errors of defined routers and services. E.g., if you have a router catching Host(`myRouter.localhost`) and you try to get:

http://myRouter.localhost/this/path/does/not/exist

then, the ErrorPage middleware can handle the wrong path serving a custom 404 page. However, it seems there is no way to exploit it to manage requests to undefined routers/services. E.g., if you try to get:

http://not-exists.localhost/foo/bar

Traefik still serves its own 404 page not found page.

My solution

In a nutshell: I define a low-priority catchall router rule that kicks in only if other routers for defined services can't handle the requests. Then, those unknown requests are redirected to a custom 404 page served by a specific service.

This is how it works:

docker-compose.yml

version: '3.7'
services:
  traefik:
    image: traefik:v2.2.5
    container_name: traefik
    security_opt:
      - no-new-privileges:true
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    ports:
      - 80:80
    command:
      # Activate dashboard.
      - --api.dashboard=true
      # Enable Docker backend with default settings.
      - --providers.docker=true
      # Do not expose containers by default.
      - --providers.docker.exposedbydefault=false
      # Default Docker network used.
      - --providers.docker.network=proxy
      # 80 (i.e., name = webinsercure)
      - --entrypoints.webinsecure.address=:80
    networks:
      - proxy

    labels:
      traefik.enable: true

      # Dashboard
      traefik.http.routers.traefik.rule: Host(`traefik.localhost`)
      traefik.http.routers.traefik.service: api@internal
      traefik.http.routers.traefik.entrypoints: webinsecure

      # Custom Error Page for undefined services
      # Get all request (but with a very low priority in order to not overlap with other rules
      traefik.http.routers.error-router.rule: "HostRegexp(`{host:.+}`)"
      traefik.http.routers.error-router.entrypoints: webinsecure
      traefik.http.routers.error-router.priority: 1
      # attach the middleware tocustom404 to this router
      traefik.http.routers.error-router.middlewares: tocustom404

      # tocustom404: redirect all to http://error.localhost/404.html
      traefik.http.middlewares.tocustom404.redirectregex.regex: ^(.*)
      traefik.http.middlewares.tocustom404.redirectregex.replacement: http://error.localhost/404.html

  nginxError:
    image: nginx:latest
    volumes:
      # here there is the page 404.html
      - ./html:/usr/share/nginx/html
    networks:
      - proxy
    labels:
      traefik.enable: true
      traefik.http.routers.defaultnginx.rule: Host(`error.localhost`)
      traefik.http.routers.defaultnginx.entrypoints: webinsecure


  defined-app:
    image: containous/whoami
    networks:
      - proxy
    labels:
      traefik.enable: true
      traefik.http.routers.my-test-app.rule: Host(`test.localhost`)
      traefik.http.routers.my-test-app.entrypoints: webinsecure

networks:
  proxy:
    external: true 

I just wanna share it with the community for comments and suggestions. I don't know if it is the Traefik-onic way or if there is a best practice to do this. But, it seems working fine and I have not found alternatives so far.

[floatingpurr]

Back again to this topic, cause I realized that the solution I described in the previous post is sub-optimal. Indeed, that was just a redirection but not a proper handling of not found errors. To improve that, the solution is attaching the ErrorPage middleware to the low-priority catchall router (Yes, I was wrong, you can definitely do that!).

Ok, let's recap:

In a nutshell: we are gonna define a low-priority catchall router rule that kicks in only if other routers for defined services can’t handle the request. Then, such an unknown request is handled by the ErrorPage middleware that tells Nginx to serve the error page.

This is how it works:

docker-compose.yml

version: '3.7'

services:
  # A cool reverse-proxy / load balancer
  traefik:
    # The official v2 Traefik docker image
    image: traefik:v2.2.7
    container_name: traefik
    security_opt:
      - no-new-privileges:true
    restart: always
    volumes:
      # So that Traefik can listen to the Docker events
      - /var/run/docker.sock:/var/run/docker.sock:ro
    ports:
      # http
      - 80:80
    command:
      ###########################################
      #   Static Configuration harnessing CLI   #
      ###########################################
      # Activate dashboard.
      - --api.dashboard=true

      # Enable Docker backend with default settings.
      - --providers.docker=true
      # Do not expose containers by default.
      - --providers.docker.exposedbydefault=false
      # Default Docker network used.
      - --providers.docker.network=proxy

      # --entrypoints.<name>.address for ports
      # 80 (i.e., name = webinsercure)
      - --entrypoints.webinsecure.address=:80

    networks:
      # This is the network over which Traefik communicates with other containers.
      - proxy

    labels:
      ################################################
      #   Dynamic configuration with Docker Labels   #
      ################################################
      # You can tell Traefik to consider (or not) this container by setting traefik.enable to true or false.
      # We need it for the dashboard
      traefik.enable: true

      # Dashboard
      traefik.http.routers.traefik.rule: Host(`traefik.localhost`)
      traefik.http.routers.traefik.service: api@internal
      traefik.http.routers.traefik.entrypoints: webinsecure


  # The error pages server
  nginxError:
    image: nginx:latest
    volumes:
      - ./error-pages:/usr/share/nginx/error-pages # error pages
      - ./nginx/default.conf:/etc/nginx/conf.d/default.conf # default configuration
    networks:
      # This is the network over which Traefik communicates with other containers.
      - proxy
    labels:
      traefik.enable: true

      traefik.http.routers.error-router.rule: HostRegexp(`{host:.+}`)
      traefik.http.routers.error-router.priority: 1
      traefik.http.routers.error-router.entrypoints: webinsecure
      traefik.http.routers.error-router.middlewares: error-pages-middleware
  
      traefik.http.middlewares.error-pages-middleware.errors.status: 400-599
      traefik.http.middlewares.error-pages-middleware.errors.service: error-pages-service
      traefik.http.middlewares.error-pages-middleware.errors.query: /{status}.html

      traefik.http.services.error-pages-service.loadbalancer.server.port: 80


  # A defined service
  my-test-app:
    image: containous/whoami
    networks:
      # This is the network over which Traefik communicates with other containers.
      - proxy
    labels:
      traefik.enable: true
      traefik.http.routers.my-test-app.rule: Host(`test.localhost`)
      traefik.http.routers.my-test-app.entrypoints: webinsecure
      traefik.http.services.my-test-app.loadbalancer.server.port: 80


networks:
  proxy:
    external: true

You need also to confiugure Nginx to serve error pages

default.conf

server {
    listen       80;
    server_name  localhost;

    error_page  404    /404.html;
    # other error pages here:
    # error_page  403    /403.html;

    location / {
        root  /usr/share/nginx/error-pages;
        internal;
    }
}

Read more:

• Here you can find static error pages (ready for Traefik) https://github.com/tarampampam/error-pages
• I wrote down some other examples here: https://imandrea.me/blog/traefik-custom-404/

[anarion]

Hello @floatingpurr and others.

I am struggling with getting the catch-all router/service working. I have read the above and Customzing a global 404 page in Traefik v.2 | Hi I’m Andrea! and to me it is fine. The issue though is, that once a request for non-existing domain hits Traefik, it doesn't generate 404 or go through error middleware as I would expect, but forwards the request to the error service directly. And only the response from that service generates 403 or 404, hits the middleware, goes back to the error service.

My docker is set-up in the following way (through Ansible):

- name: Traefik Docker Container
  docker_container:
    name: traefik
    image: "{{ traefik_docker_image }}"
    pull: true
    network_mode: host
    volumes:
      - "{{ traefik_data_directory }}/traefik.toml:/etc/traefik/traefik.toml:ro"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    ports:
      - "{{ traefik_port_http }}:80"
      - "{{ traefik_port_https }}:443"
      - "{{ traefik_port_ui }}:8080"
    labels:
      traefik.enable: "true"
      traefik.http.routers.traefik-secure.entrypoints: "websecure"
      traefik.http.routers.traefik-secure.rule: "Host(`traefik.domain.com`)"
      traefik.http.routers.traefik-secure.service: "api@internal"
      traefik.http.routers.traefik-secure.middlewares: "traefik-whitelist"
      traefik.http.routers.traefik.entrypoints: "web"
      traefik.http.routers.traefik.rule: "Host(`traefik.domain.com`)"
      traefik.http.routers.traefik.service: "api@internal"
      traefik.http.routers.traefik.middlewares: "traefik-whitelist"
      traefik.http.services.traefik-secure.loadbalancer.server.port: "{{ traefik_port_ui }}"
      traefik.http.services.traefik.loadbalancer.server.port: "{{ traefik_port_ui }}"
      traefik.http.middlewares.traefik-whitelist.ipwhitelist.ipstrategy.depth: "1"
      traefik.http.middlewares.traefik-whitelist.ipwhitelist.sourcerange: "127.0.0.1/32, 192.168.0.0/16"
    restart_policy: unless-stopped
    memory: 1g

- name: Nginx for Errors Docker Container
  docker_container:
    name: nginxErrors
    image: nginx:latest
    pull: true
    network_mode: bridge
    volumes:
      - "{{ traefik_data_directory }}/error-pages:/usr/share/nginx/error-pages"
      - "{{ traefik_data_directory }}/nginx/default.conf:/etc/nginx/conf.d/default.conf"
    labels:
      traefik.enable: "true"
      traefik.http.routers.error-router.rule: "HostRegexp(`{host:.+}`)"
      traefik.http.routers.error-router.priority: "1"
      traefik.http.routers.error-router.entrypoints: "web,websecure"
      traefik.http.routers.error-router.middlewares: "error-pages-middleware"
      traefik.http.middlewares.error-pages-middleware.errors.status: "400-599"
      traefik.http.middlewares.error-pages-middleware.errors.service: "error-pages-service"
      traefik.http.middlewares.error-pages-middleware.errors.query: /{status}.html
      traefik.http.services.error-pages-service.loadbalancer.server.port: "80"
    restart_policy: unless-stopped

The Nginx under error-pages-service is very simple:

server {
    listen       80;
    server_name  localhost;

    error_page 401 /401.html;
    error_page 403 /403.html;
    error_page 404 /404.html;
    error_page 405 /405.html;
    error_page 407 /407.html;
    error_page 408 /408.html;
    error_page 409 /409.html;
    error_page 410 /410.html;
    error_page 411 /411.html;
    error_page 412 /412.html;
    error_page 413 /413.html;
    error_page 416 /411.html;
    error_page 418 /412.html;
    error_page 421 /421.html;
    error_page 429 /429.html;
    error_page 500 /500.html;
    error_page 502 /502.html;
    error_page 503 /503.html;
    error_page 504 /504.html;
    error_page 505 /505.html;

    location / {
        root  /usr/share/nginx/error-pages;
        #internal;
    }
}

And this is traefik debug log:

time="2021-01-07T13:08:17Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.0\",\"ProtoMajor\":1,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,en;q=0.5\"],\"Connection\":[\"close\"],\"Dnt\":[\"1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0\"],\"X-Client-Verify\":[\"NONE\"],\"X-Forwarded-For\":[\"192.168.2.10\"],\"X-Forwarded-Host\":[\"non-existent-domain.domain.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"nas\"],\"X-Real-Ip\":[\"192.168.2.10\"],\"X-Tls-Cipher\":[\"TLS_AES_256_GCM_SHA384\"],\"X-Tls-Client-Intercepted\":[\"Unknown\"],\"X-Tls-Protocol\":[\"TLSv1.3\"],\"X-Tls-Sni-Host\":[\"non-existent-domain.domain.com\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"non-existent-domain.domain.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.2.1:64380\",\"RequestURI\":\"/\",\"TLS\":null}"
time="2021-01-07T13:08:17Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.0\",\"ProtoMajor\":1,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,en;q=0.5\"],\"Connection\":[\"close\"],\"Dnt\":[\"1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0\"],\"X-Client-Verify\":[\"NONE\"],\"X-Forwarded-For\":[\"192.168.2.10\"],\"X-Forwarded-Host\":[\"non-existent-domain.domain.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"nas\"],\"X-Real-Ip\":[\"192.168.2.10\"],\"X-Tls-Cipher\":[\"TLS_AES_256_GCM_SHA384\"],\"X-Tls-Client-Intercepted\":[\"Unknown\"],\"X-Tls-Protocol\":[\"TLSv1.3\"],\"X-Tls-Sni-Host\":[\"non-existent-domain.domain.com\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"non-existent-domain.domain.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.2.1:64380\",\"RequestURI\":\"/\",\"TLS\":null}" ForwardURL="http://172.17.0.43:80"
time="2021-01-07T13:08:17Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.0\",\"ProtoMajor\":1,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,en;q=0.5\"],\"Connection\":[\"close\"],\"Dnt\":[\"1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0\"],\"X-Client-Verify\":[\"NONE\"],\"X-Forwarded-For\":[\"192.168.2.10\"],\"X-Forwarded-Host\":[\"non-existent-domain.domain.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"nas\"],\"X-Real-Ip\":[\"192.168.2.10\"],\"X-Tls-Cipher\":[\"TLS_AES_256_GCM_SHA384\"],\"X-Tls-Client-Intercepted\":[\"Unknown\"],\"X-Tls-Protocol\":[\"TLSv1.3\"],\"X-Tls-Sni-Host\":[\"non-existent-domain.domain.com\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"non-existent-domain.domain.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.2.1:64380\",\"RequestURI\":\"/\",\"TLS\":null}"
time="2021-01-07T13:08:17Z" level=error msg="Caught HTTP Status Code 403, returning error page" middlewareName=error-pages-middleware@docker middlewareType=customError
time="2021-01-07T13:08:17Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"0.0.0.0\",\"Path\":\"/403.html\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,en;q=0.5\"],\"Connection\":[\"close\"],\"Dnt\":[\"1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0\"],\"X-Client-Verify\":[\"NONE\"],\"X-Forwarded-For\":[\"192.168.2.10\"],\"X-Forwarded-Host\":[\"non-existent-domain.domain.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"nas\"],\"X-Real-Ip\":[\"192.168.2.10\"],\"X-Tls-Cipher\":[\"TLS_AES_256_GCM_SHA384\"],\"X-Tls-Client-Intercepted\":[\"Unknown\"],\"X-Tls-Protocol\":[\"TLSv1.3\"],\"X-Tls-Sni-Host\":[\"non-existent-domain.domain.com\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"0.0.0.0\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"\",\"RequestURI\":\"/403.html\",\"TLS\":null}"
time="2021-01-07T13:08:17Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"0.0.0.0\",\"Path\":\"/403.html\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,en;q=0.5\"],\"Connection\":[\"close\"],\"Dnt\":[\"1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0\"],\"X-Client-Verify\":[\"NONE\"],\"X-Forwarded-For\":[\"192.168.2.10\"],\"X-Forwarded-Host\":[\"non-existent-domain.domain.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"nas\"],\"X-Real-Ip\":[\"192.168.2.10\"],\"X-Tls-Cipher\":[\"TLS_AES_256_GCM_SHA384\"],\"X-Tls-Client-Intercepted\":[\"Unknown\"],\"X-Tls-Protocol\":[\"TLSv1.3\"],\"X-Tls-Sni-Host\":[\"non-existent-domain.domain.com\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"0.0.0.0\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"\",\"RequestURI\":\"/403.html\",\"TLS\":null}" ForwardURL="http://172.17.0.43:80"
time="2021-01-07T13:08:17Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"0.0.0.0\",\"Path\":\"/403.html\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,en;q=0.5\"],\"Connection\":[\"close\"],\"Dnt\":[\"1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0\"],\"X-Client-Verify\":[\"NONE\"],\"X-Forwarded-For\":[\"192.168.2.10\"],\"X-Forwarded-Host\":[\"non-existent-domain.domain.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"nas\"],\"X-Real-Ip\":[\"192.168.2.10\"],\"X-Tls-Cipher\":[\"TLS_AES_256_GCM_SHA384\"],\"X-Tls-Client-Intercepted\":[\"Unknown\"],\"X-Tls-Protocol\":[\"TLSv1.3\"],\"X-Tls-Sni-Host\":[\"non-existent-domain.domain.com\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"0.0.0.0\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"\",\"RequestURI\":\"/403.html\",\"TLS\":null}"

I removed the 'internal' directive from Nginx as with it I was getting 404 all the time instead of actual error page - something to check independently.

With the above the request hits Nginx and it responds:

172.17.0.1 - - [07/Jan/2021:13:08:17 +0000] "GET / HTTP/1.1" 403 1459 "-" "Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0" "192.168.2.10, 192.168.2.1"
172.17.0.1 - - [07/Jan/2021:13:08:17 +0000] "GET /403.html HTTP/1.1" 200 1459 "-" "Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0" "192.168.2.10"
2021/01/07 13:08:17 [error] 28#28: *8 directory index of "/usr/share/nginx/error-pages/" is forbidden, client: 172.17.0.1, server: localhost, request: "GET / HTTP/1.1", host: "non-existent-domain.domain.com"

So what am I missing?

I have an app that is working fine - the request first hits the ipwhitelist, traefik generates 403 and that 403 is then send to Nginx error service.

[floatingpurr]

Hey, did you manage to fix that?

I cannot go over your configuration line by line, but you need to make sure that all the requests that do not match any router are caught by your low-priority catchall router. Then attach your error-pages-middleware to it.

If you want, you can start from the working example I posted. Then, you can add incrementally your other labels in order to troubleshoot your config issue.

[anarion]

No, unfortunately not.
I have exactly the same labels as you.
I installed https://github.com/tarampampam/error-pages and the behaviour is the same - the router first sends the request to the service, it bounces back with an error and only then goes to the service again with correct error URI.
I'm beginning to think that this is actually expected behavior...

Thanks for the response anyway.

[floatingpurr]

Sorry for the late reply. Can you kindly try with this docker-compose file?

[anarion]

I have switched to tarampampam/error-pages and stopped investigating. I checked now and it seems to work fine - gives me 404 when there is an unknown domain, and 403 when there is no match in the ipwhitelist. So that's fine, although I can't bind mount (and thus edit) those error pages there - at least I don't know how.

[SoulAssassin]

Hey guys!

I think I found in this topic what I need. But, I'm looking for something like global redirect also, if subdomain is not exist. Anybody done it or make some steps on it?

Like non-existent-subdomain.mydomain.com -> google.com

[Direct]

Did you found any solution for this ? I hope to be able to do the same.

[ScreN]

Same problem. If you write subdomain.domain.com in the browser, the browser returns the usual 404, instead of the beautiful 404 from tarampampam.

In fact, I'm not really sure about the safety of using the container from tarampampam.