Details on 404 request?

I am setting up traefik and am looking to see how I can get more details from the logs when a request returns 404? Is there any way I can see the details of the incmming request (headers, referrer, client ip, etc)? I am trying to track down when requests for hosts are failing -- the container shows up in the traefik dashoard, the host rules look correct, and the back end container is up and running.

Any suggestions on how to trace the issue?

Note: I have enabled debug logging and it doesn't seem to how anything for 404 requests (does that sound right)?

1 Like

If you enable the access log you can see which router is handling the request.

If you are still stuck you can post your config and someone is likely to help.

Thanks -- I did enable it and all I am seeing is

Browser URL: https://nextcloud-phpmyadmin.home.local/

access.log

172.18.0.1 - - [14/May/2020:12:56:58 +0000] "GET / HTTP/2.0" 404 19 "-" "-" 6 "-" "-" 0ms

traefik.log -- no entry for the request

For my configuration here is what I have

traefik

version: "3.7"

services:

  traefik:
    
    container_name: traefik
    image: traefik:latest
    
    env_file: .env
    volumes:
      - ./data/certs:/certs
      - ./data/traefik:/etc/traefik/dynamic
      - ./data/letsencrypt:/letsencrypt
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - public
      - data
    ports:
      - 80:80
      - 443:443
      - 8080:8080
    command:
      - "--log.level=DEBUG"
      - "--log.filePath=/logs/traefik.log"
      - "--log.format=json"
      - "--accesslog=true"
      - "--accesslog.filepath=/logs/access.log"
      - "--api.dashboard=true"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.endpoint=unix:///var/run/docker.sock"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.docker.useBindPortIP=true"
      - "--providers.file.directory=/etc/traefik/dynamic"
      - "--providers.file.watch=true"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
      - "--entrypoints.web.http.redirections.entryPoint.scheme=https"
      - "--certificatesresolvers.secureresolver.acme.httpchallenge=true"
      - "--certificatesresolvers.secureresolver.acme.httpchallenge.entrypoint=web"
      - "--certificatesresolvers.secureresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
      - "--certificatesresolvers.secureresolver.acme.email=certs@eye-catcher.com"
      - "--certificatesresolvers.secureresolver.acme.storage=/letsencrypt/acme.json"
      - "--serverstransport.insecureskipverify=true"
    security_opt:
      - no-new-privileges:true
    labels:
      - "com.centurylinklabs.watchtower.enable=true"
      - "traefik.enable=true"
      - "traefik.http.routers.dashboard.entryPoints=traefik"
      - "traefik.http.routers.dashboard.rule=PathPrefix(`/api`) || PathPrefix(`/dashboard`)"
      - "treafik.http.routers.dashboard.service=api@internal"
    restart: always

Next cloud

version: "3.7"

services:

  nextcloud:
    # https://hub.docker.com/r/linuxserver/nextcloud
    container_name: nextcloud
    image: linuxserver/nextcloud
    env_file: .env
    environment:
      - MYSQL_HOST=nextclouddb
      - MYSQL_USER=nextcloud
      - MYSQL_PASSWORD=xxxxxx
      - MYSQL_DATABASE=nextcloud
      - NEXTCLOUD_ADMIN_USER=nextcloudadmin
      - NEXTCLOUD_ADMIN_PASSWORD=xxxxxx
    volumes:
      - ./data/nextcloud:/config
      - ./data/db:/data
      - ./init/custom-cont-init.d:/config/custom-cont-init.d
      - ./init/config:/root/config.tmp
    networks:
      - data
    depends_on:
      - nextclouddb
    security_opt:
      - no-new-privileges:true
    labels:
      - "com.centurylinklabs.watchtower.enable=true"
      - "traefik.enable=true"
      - "traefik.docker.network=data"
      - "traefik.http.routers.nextcloud.entrypoints=websecure"
      - "traefik.http.routers.nextcloud.rule=Host(`nextcloud.home.local`)" 
      - "traefik.http.routers.nextcloud.tls=true"
      - "traefik.http.routers.nextcloud.service=nextcloud"
      - "traefik.http.services.nextcloud.loadbalancer.server.port=443" 
      - "traefik.http.services.nextcloud.loadbalancer.server.scheme=https"
    restart: unless-stopped

  nextclouddb:
    # https://hub.docker.com/r/linuxserver/mariadb
    container_name: nextclouddb
    image: linuxserver/mariadb:latest
    hostname: nextclouddb
    env_file: .env
    environment:
      - MYSQL_ROOT_PASSWORD=xxxxxx
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
      - MYSQL_PASSWORD=xxxxxx
    volumes:
      - ./data/db:/config
    networks:
      - data
    security_opt:
      - no-new-privileges:true
    labels:
      - "com.centurylinklabs.watchtower.enable=true"
    restart: unless-stopped

  nextclouddb-phpmyadmin:
    # https://hub.docker.com/r/phpmyadmin/phpmyadmin
    container_name: nextclouddb-phpmyadmin
    image: phpmyadmin/phpmyadmin:latest
    env_file: .env
    environment:
      - PMA_ABSOLUTE_URI=https://nextcloud-phpmyadmin.home.local
      - PMA_ARBITRARY=1
      - PMA_HOST=nextclouddb
      - PMA_USER=nextcloud
      - PMA_PASSWORD=xxxxxx
      - MYSQL_ROOT_PASSWORD=xxxxxx
      - MYSQL_USER=nextcloud
      - MYSQL_PASSWORD=blueprint-title-multimeter
    volumes:
      - ./data/myadmin:/var/lib/mysql
    networks:
      - data
    depends_on:
      - nextclouddb
    security_opt:
      - no-new-privileges:true
    labels:
      - "com.centurylinklabs.watchtower.enable=true"
      - "traefik.enable=true"
      - "traefik.docker.network=data"
      - "traefik.http.routers.nextclouddb-phpmyadmin.entrypoints=websecure"
      - "traefik.http.routers.nextclouddb-phpmyadmin.rule=Host(`nextcloud-phpmyadmin.home.local`)" 
      - "traefik.http.routers.nextclouddb-phpmyadmin.tls=false"
      - "traefik.http.routers.nextclouddb-phpmyadmin.service=nextclouddb-phpmyadmin"
      - "traefik.http.services.nextclouddb-phpmyadmin.loadbalancer.server.port=80" 
      - "traefik.http.services.nextclouddb-phpmyadmin.loadbalancer.server.scheme=http"
    restart: unless-stopped

Accessing https://nextcloud.home.local/ works just fine. So the question is - why is one working and not the other?

Also just to confirm -- both routes and services are showing up in the Traefik dashboard as health.

You have - "traefik.http.routers.nextclouddb-phpmyadmin.tls=false" but you are accessing on https.

So there is no rule match. You get 404 and no router is identified in the log.

Hmm, I though that was for the BACKEND service -- my public access is secure, however the backend service is over http and port 80. Do i have that wrong? I think I made the assumptions (I know bad idea) that since the entrypoint was secure I was all good.

So that leads me to a question -- why have the entrypoint and the tls route setting separate? is there any scenerio where you have 1 backend service using an entrypoint with TLS and another usnig the same entrypoint without ?

@cakiwi -- that did the trick. Thanks!

@cakiwi - I did have one other question. Any idea how to get this information logged -- I sort of expected to see something in the traefik log that indicated an incoming request was made and no route was found.

A 404 means: no route found :wink:

I've learned that a routed request will have the name of the router in the access log "nextcloud@docker". With an unmatched route that field is a -

<remote_IP_address> - <client_user_name_if_available> [<timestamp>] "<request_method> <request_path> <request_protocol>" <origin_server_HTTP_status> <origin_server_content_size> "<request_referrer>" "<request_user_agent>" <number_of_requests_received_since_Traefik_started> "<Traefik_router_name>" "<Traefik_server_URL>" <request_duration_in_ms>ms

https://docs.traefik.io/observability/access-logs/#format

No it doesn't that indicates resource not found. You can have a perfectly working router and still get a 404 from the backend.

Ah! So you're reffering to if a backend is returning a 404. Okay, I did missunderstood you then :slight_smile:

No. I am am saying it is erroneous to say A 404 means: no route found as a blanket statement both in relation to traefik and http codes in general.

1 Like