Getting 401 or 404 HTTP errors for every service

Traefik Traefik v2
1

Hi there,

Despite reading carefully all the v2 doc, I'm stuck since more than one week on a simple problem: Traefik cannot contact services. I will detail here my complete setup.

SETUP

1. Domain & DNS
I have a domain called domain.fr. With this domain, I added the following DNS records:

  • (CNAME) www.domain.fr -> domain.fr
  • (A) domain.fr -> WAN IP of my FAI box [router] (e.g.: 81.63.80.102)
  • (A) admin.domain.fr -> WAN IP of my FAI box [router] (e.g.: 81.63.80.102)
  • (A) service1.domain.fr -> WAN IP of my FAI box [router] (e.g.: 81.63.80.102)
  • (A) service2.domain.fr -> WAN IP of my FAI box [router] (e.g.: 81.63.80.102)

2. LAN: FAI router configuration

  • All internet requests from port (TCP) 80/443 are redirected to the port (TCP) 80/443 of a local fixed IPv4 -> 192.168.0.100.

3. LAN: hardware configuration

  • 192.168.0.100 -> (DMZ) Raspberry Pi (Raspbian). Runs Traefik and catch all HTTP/HTTPS requests (80/443).
  • 192.168.0.110 -> (NAS) Synology NAS. Runs all services I want. I use Docker containers for almost every service. The only ONE service not using Docker is the default homepage for my domain. This is a static website served by a local Apache instance managed in WebStation, a DSM application (Synology).

4. Traefik routing logic & configuration files

Routing logic: 
domain.fr -> default static homepage
admin.domain.fr/service1 -> administration service n°1
admin.domain.fr/service1 -> ...
service1.domain.fr -> user service n°1
service2.domain.fr -> ...
docker-compose.yml 
# /app/traefik/docker-compose.yml

version: "3.3"

services:

  traefik:
    image: traefik:v2.0
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    ports:
      - 80:80
#      - 443:443
#      - 8080:8080
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ${PWD}/conf/:/etc/traefik/
      - ${PWD}/letsencrypt/:/letsencrypt/
      - ${PWD}/log/:/var/log/traefik/
    networks:
      - proxy

networks:
  proxy:
    external: true
traefik.yml 
# /app/traefik/conf/traefik.yml

## Static configuration
#
# @see https://docs.traefik.io/v2.0/getting-started/configuration-overview/#the-static-configuration


# Global
#
global:
  sendAnonymousUsage: true


# Ping
#
#ping:
#  entryPoint: http


# Servers Transport
serversTransport:
  # Disables SSL certificate verification between Traefik instance and the backends
  # @see: https://docs.traefik.io/v2.0/routing/overview/#insecureskipverify
  insecureSkipVerify: true


# Logs
#
# @see   https://docs.traefik.io/v2.0/observability/logs/
# @see   https://docs.traefik.io/v2.0/observability/access-logs/
# @info
#        # Log > Levels
#        DEBUG, INFO, WARN, [ERROR], FATAL, PANIC
#
#        # Accesslog > bufferingSize
#        Number of buffered lines to write in the file
log:
  level: DEBUG
  filePath: "/var/log/traefik/errors.log"
  format: json

accessLog:
  filePath: "/var/log/traefik/access.log"
  bufferingSize: 200
  format: json


# Traefik Dashboard
#
# Central place that visualize the current active routes handled by Traefik
#
# @see  https://docs.traefik.io/v2.0/operations/api/
# DEV
api:
  dashboard: true
#  insecure: true
  debug: true


# Providers
#
# Configuration discovery in Traefik is achieved through Providers. The providers
# are existing infrastructure components, whether orchestrators, container engines, cloud providers, or key-value stores. The idea is that Traefik will query the providers' API in order to find relevant information about routing, and each time Traefik detects a change, it dynamically updates the routes.
providers:
  # Docker
  # @see https://docs.traefik.io/providers/docker/#provider-configuration-options
  docker:
    endpoint: "unix:///var/run/docker.sock"
    network: proxy
    exposedByDefault: false
#    defaultRule: foobar
#    tls:
#      ca: foobar
#      caOptional: true
#      cert: foobar
#      key: foobar
#      insecureSkipVerify: true
  # Dynamic configuration file
  file:
    filename: "/etc/traefik/dynamic.yml"


# EntryPoints
#
# Define the port which will receive the HTTP/TCP requests
#
# @see https://docs.traefik.io/v2.0/routing/entrypoints/
entryPoints:
  http:
    address: ":80"
#  https:
#    address: ":443"


# Certificates Resolvers
#
#certificatesResolvers:
  # Let's Encrypt HTTP resolver
  #
  # @see https://docs.traefik.io/https/acme/
  # @see https://docs.traefik.io/user-guides/docker-compose/acme-http/
#  letsEncryptHTTPCertificateResolver:
#    acme:
#      email: "postmaster@domain.fr"
#      storage: "/letsencrypt/acme.json"
#      httpChallenge:
#        entryPoint: http
dynamic.yml 
# /app/traefik/conf/dynamic.yml

## Dynamic configuration
#
# @see https://docs.traefik.io/v2.0/getting-started/configuration-overview/#the-dynamic-configuration


# HTTP
#
# @info Every router/middleware/service will only focus on HTTP protocol.
http:


  # Routers
  #
  # Connecting incoming requests to the services that can handle them
  #
  # @see https://docs.traefik.io/v2.0/routing/routers/
  routers:

    # URL: domain.fr
    home-insecure:
      entryPoints:
      - http
      rule: Host(`domain.fr`)
      service: homepage


    # URL: admin.domain.fr/nas/nas1
    nas-nas1-insecure:
      entryPoints:
      - http
      rule: Host(`admin.domain.fr`) && PathPrefix(`/nas/nas1`)
      service: nas-nas1-webui

    # URL: service1.domain.fr
    service1-insecure:
      entryPoints:
      - http
      rule: Host(`service1.domain.fr`)
      service: service1-webui

    # URL: admin.domain.fr/proxy
    traefik-insecure:
      entryPoints:
      - http
      rule: Host(`admin.domain.fr`) && PathPrefix(`/proxy`)
      service: api@internal


  # Middlewares
  #
  # Mean of tweaking the requests (routers) before they are sent to the services
  #
  # @see https://docs.traefik.io/v2.0/middlewares/overview/
  middlewares:

    # Admin authentication
    admin-auth:
      basicAuth:
        users:
        # test purpose -> admin:admin
        - "admin:$apr1$hzSvASf9$nuc/a9UN5AVEh6Ceg.Eqi0"

    # Force HTTPS
    http-to-https:
      redirectScheme:
        scheme: https

    # Basic security headers
    default-headers:
      headers:
        frameDeny: true
        sslRedirect: true
        browserXssFilter: true
        contentTypeNosniff: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsPreload: true

    # Allow only inernal IP addresses
    lan-ip-whitelist:
      ipWhiteList:
        sourceRange:
        - "10.0.0.0/24"
        - "192.168.0.0/16"
        - "172.0.0.0/8"

    # Secured (WAN)
    secured:
      chain:
        middlewares:
        - http-to-https
        - default-headers

    # Secured (MAN)
    secured-local:
      chain:
        middlewares:
        - secured
        - lan-ip-whitelist

  # Services
  #
  # Configuring how to reach the actual services that will eventually handle the incoming requests
  #
  # @see https://docs.traefik.io/v2.0/routing/services/
  services:

    # Homepage
    homepage:
      loadBalancer:
        servers:
          - url: "http://192.168.0.110:80"
        healthCheck:
          path: /_health
          interval: "10s"
          timeout: "3s"

    # NAS (nas1) - Web UI
    nas-nas1-webui:
      loadBalancer:
        servers:
          - url: "https://192.168.0.110:5001"
        healthCheck:
          path: /_health
          interval: "10s"
          timeout: "3s"

    # Torrent - Web UI
    service1-webui:
      loadBalancer:
        servers:
          - url: "http://192.168.0.110:17010"
        healthCheck:
          path: /_health
          interval: "10s"
          timeout: "3s"

5. Traefik logs

{"level":"info","msg":"Traefik version 2.0.1 built on 2019-09-26T16:25:50Z","time":"2019-10-07T16:34:31+02:00"}
{"level":"debug","msg":"Static configuration loaded {\"global\":{\"checkNewVersion\":true,\"sendAnonymousUsage\":true},\"serversTransport\":{\"insecureSkipVerify\":true,\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"http\":{\"address\":\":80\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{}}},\"providers\":{\"providersThrottleDuration\":2000000000,\"docker\":{\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"network\":\"proxy\",\"swarmModeRefreshSeconds\":15000000000},\"file\":{\"watch\":true,\"filename\":\"/etc/traefik/dynamic.yml\"}},\"api\":{\"dashboard\":true,\"debug\":true},\"log\":{\"level\":\"DEBUG\",\"filePath\":\"/var/log/traefik/errors.log\",\"format\":\"json\"},\"accessLog\":{\"filePath\":\"/var/log/traefik/access.log\",\"format\":\"json\",\"filters\":{},\"fields\":{\"defaultMode\":\"keep\",\"headers\":{\"defaultMode\":\"drop\"}},\"bufferingSize\":200}}","time":"2019-10-07T16:34:31+02:00"}
{"level":"info","msg":"Stats collection is enabled.","time":"2019-10-07T16:34:31+02:00"}
{"level":"info","msg":"Many thanks for contributing to Traefik's improvement by allowing us to receive anonymous information from your configuration.","time":"2019-10-07T16:34:31+02:00"}
{"level":"info","msg":"Help us improve Traefik by leaving this feature on :)","time":"2019-10-07T16:34:31+02:00"}
{"level":"info","msg":"More details on: https://docs.traefik.io/v2.0/contributing/data-collection/","time":"2019-10-07T16:34:31+02:00"}
{"level":"debug","msg":"No default certificate, generating one","time":"2019-10-07T16:34:31+02:00"}
{"entryPointName":"http","level":"debug","msg":"Start TCP Server","time":"2019-10-07T16:34:42+02:00"}
{"level":"info","msg":"Starting provider aggregator.ProviderAggregator {}","time":"2019-10-07T16:34:42+02:00"}
{"level":"info","msg":"Starting provider *file.Provider {\"watch\":true,\"filename\":\"/etc/traefik/dynamic.yml\"}","time":"2019-10-07T16:34:42+02:00"}
{"level":"info","msg":"Starting provider *docker.Provider {\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"network\":\"proxy\",\"swarmModeRefreshSeconds\":15000000000}","time":"2019-10-07T16:34:42+02:00"}
{"level":"debug","msg":"Configuration received from provider file: {\"http\":{\"routers\":{\"home-insecure\":{\"entryPoints\":[\"http\"],\"service\":\"homepage\",\"rule\":\"Host(`domain.fr`)\"},\"nas-nas1-insecure\":{\"entryPoints\":[\"http\"],\"service\":\"nas-nas1-webui\",\"rule\":\"Host(`admin.domain.fr`) \\u0026\\u0026 PathPrefix(`/nas/nas1`)\"},\"service1-insecure\":{\"entryPoints\":[\"http\"],\"service\":\"service1-webui\",\"rule\":\"Host(`service1.domain.fr`)\"},\"traefik-insecure\":{\"entryPoints\":[\"http\"],\"service\":\"api@internal\",\"rule\":\"Host(`admin.domain.fr`) \\u0026\\u0026 PathPrefix(`/proxy`)\"}},\"middlewares\":{\"admin-auth\":{\"basicAuth\":{\"users\":[\"izin:$apr1$QEX3Fz.s$c5Vi5EtQnrCAgVNUf67Hp0\",\"admin:$apr1$hzSvASf9$nuc/a9UN5AVEh6Ceg.Eqi0\"]}},\"default-headers\":{\"headers\":{\"sslRedirect\":true,\"stsIncludeSubdomains\":true,\"stsPreload\":true,\"forceSTSHeader\":true,\"frameDeny\":true,\"contentTypeNosniff\":true,\"browserXssFilter\":true}},\"lan-ip-whitelist\":{\"ipWhiteList\":{\"sourceRange\":[\"10.0.0.0/24\",\"192.168.0.0/16\",\"172.0.0.0/8\"]}},\"secured\":{\"chain\":{\"middlewares\":[\"default-headers\"]}},\"secured-local\":{\"chain\":{\"middlewares\":[\"secured\",\"lan-ip-whitelist\"]}}},\"services\":{\"homepage\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://192.168.0.110:80\"}],\"healthCheck\":{\"path\":\"/_health\",\"interval\":\"10s\",\"timeout\":\"3s\"},\"passHostHeader\":false}},\"nas-nas1-webui\":{\"loadBalancer\":{\"servers\":[{\"url\":\"https://192.168.0.110:5001\"}],\"healthCheck\":{\"path\":\"/_health\",\"interval\":\"10s\",\"timeout\":\"3s\"},\"passHostHeader\":false}},\"service1-webui\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://192.168.0.110:17010\"}],\"healthCheck\":{\"path\":\"/_health\",\"interval\":\"10s\",\"timeout\":\"3s\"},\"passHostHeader\":false}}}},\"tcp\":{},\"tls\":{}}","providerName":"file","time":"2019-10-07T16:34:42+02:00"}
{"entryPointName":"http","level":"debug","middlewareName":"pipelining","middlewareType":"Pipelining","msg":"Creating middleware","routerName":"home-insecure@file","serviceName":"homepage","time":"2019-10-07T16:34:42+02:00"}
{"entryPointName":"http","level":"debug","msg":"Creating load-balancer","routerName":"home-insecure@file","serviceName":"homepage","time":"2019-10-07T16:34:42+02:00"}
{"entryPointName":"http","level":"debug","msg":"Creating server 0 http://192.168.0.110:80","routerName":"home-insecure@file","serverName":0,"serviceName":"homepage","time":"2019-10-07T16:34:42+02:00"}
{"entryPointName":"http","level":"debug","middlewareName":"tracing","middlewareType":"TracingForwarder","msg":"Added outgoing tracing middleware homepage","routerName":"home-insecure@file","time":"2019-10-07T16:34:42+02:00"}
{"entryPointName":"http","level":"debug","middlewareName":"pipelining","middlewareType":"Pipelining","msg":"Creating middleware","routerName":"nas-nas1-insecure@file","serviceName":"nas-nas1-webui","time":"2019-10-07T16:34:42+02:00"}
{"entryPointName":"http","level":"debug","msg":"Creating load-balancer","routerName":"nas-nas1-insecure@file","serviceName":"nas-nas1-webui","time":"2019-10-07T16:34:42+02:00"}
{"entryPointName":"http","level":"debug","msg":"Creating server 0 https://192.168.0.110:5001","routerName":"nas-nas1-insecure@file","serverName":0,"serviceName":"nas-nas1-webui","time":"2019-10-07T16:34:42+02:00"}
{"entryPointName":"http","level":"debug","middlewareName":"tracing","middlewareType":"TracingForwarder","msg":"Added outgoing tracing middleware nas-nas1-webui","routerName":"nas-nas1-insecure@file","time":"2019-10-07T16:34:42+02:00"}
{"entryPointName":"http","level":"debug","middlewareName":"pipelining","middlewareType":"Pipelining","msg":"Creating middleware","routerName":"service1-insecure@file","serviceName":"service1-webui","time":"2019-10-07T16:34:42+02:00"}
{"entryPointName":"http","level":"debug","msg":"Creating load-balancer","routerName":"service1-insecure@file","serviceName":"service1-webui","time":"2019-10-07T16:34:42+02:00"}
{"entryPointName":"http","level":"debug","msg":"Creating server 0 http://192.168.0.110:17010","routerName":"service1-insecure@file","serverName":0,"serviceName":"service1-webui","time":"2019-10-07T16:34:42+02:00"}
{"entryPointName":"http","level":"debug","middlewareName":"tracing","middlewareType":"TracingForwarder","msg":"Added outgoing tracing middleware service1-webui","routerName":"service1-insecure@file","time":"2019-10-07T16:34:42+02:00"}
{"entryPointName":"http","level":"debug","middlewareName":"tracing","middlewareType":"TracingForwarder","msg":"Added outgoing tracing middleware api@internal","routerName":"traefik-insecure@file","time":"2019-10-07T16:34:42+02:00"}
{"entryPointName":"http","level":"debug","middlewareName":"traefik-internal-recovery","middlewareType":"Recovery","msg":"Creating middleware","time":"2019-10-07T16:34:42+02:00"}
{"level":"debug","msg":"Setting up healthcheck for service homepage@file with [Hostname:  Headers: map[] Path: /_health Port: 0 Interval: 10s Timeout: 3s]","serviceName":"homepage@file","time":"2019-10-07T16:34:42+02:00"}
{"level":"debug","msg":"Setting up healthcheck for service nas-nas1-webui@file with [Hostname:  Headers: map[] Path: /_health Port: 0 Interval: 10s Timeout: 3s]","serviceName":"nas-nas1-webui@file","time":"2019-10-07T16:34:42+02:00"}
{"level":"debug","msg":"Setting up healthcheck for service service1-webui@file with [Hostname:  Headers: map[] Path: /_health Port: 0 Interval: 10s Timeout: 3s]","serviceName":"service1-webui@file","time":"2019-10-07T16:34:42+02:00"}
{"level":"debug","msg":"Setting up healthcheck for service homepage@file with [Hostname:  Headers: map[] Path: /_health Port: 0 Interval: 10s Timeout: 3s]","serviceName":"homepage@file","time":"2019-10-07T16:34:42+02:00"}
{"level":"debug","msg":"Initial health check for backend: \"nas-nas1-webui@file\"","time":"2019-10-07T16:34:42+02:00"}
{"level":"debug","msg":"Setting up healthcheck for service nas-nas1-webui@file with [Hostname:  Headers: map[] Path: /_health Port: 0 Interval: 10s Timeout: 3s]","serviceName":"nas-nas1-webui@file","time":"2019-10-07T16:34:42+02:00"}
{"level":"debug","msg":"Setting up healthcheck for service service1-webui@file with [Hostname:  Headers: map[] Path: /_health Port: 0 Interval: 10s Timeout: 3s]","serviceName":"service1-webui@file","time":"2019-10-07T16:34:42+02:00"}
{"level":"debug","msg":"No default certificate, generating one","time":"2019-10-07T16:34:42+02:00"}
{"level":"debug","msg":"Initial health check for backend: \"service1-webui@file\"","time":"2019-10-07T16:34:42+02:00"}
{"level":"debug","msg":"Initial health check for backend: \"nas-nas1-webui@file\"","time":"2019-10-07T16:34:42+02:00"}
{"level":"debug","msg":"Initial health check for backend: \"service1-webui@file\"","time":"2019-10-07T16:34:42+02:00"}
{"level":"debug","msg":"Initial health check for backend: \"homepage@file\"","time":"2019-10-07T16:34:42+02:00"}
{"level":"debug","msg":"Initial health check for backend: \"homepage@file\"","time":"2019-10-07T16:34:42+02:00"}
{"level":"warning","msg":"Health check failed: Remove from server list. Backend: \"service1-webui@file\" URL: \"http://192.168.0.110:17010\" Weight: 1 Reason: received error status code: 401","time":"2019-10-07T16:34:42+02:00"}
{"level":"debug","msg":"Stopping current health check goroutines of backend: service1-webui@file","time":"2019-10-07T16:34:42+02:00"}
{"level":"warning","msg":"Health check failed: Remove from server list. Backend: \"homepage@file\" URL: \"http://192.168.0.110:80\" Weight: 1 Reason: received error status code: 404","time":"2019-10-07T16:34:42+02:00"}
{"level":"debug","msg":"Stopping current health check goroutines of backend: homepage@file","time":"2019-10-07T16:34:42+02:00"}
{"level":"warning","msg":"Health check failed: Remove from server list. Backend: \"service1-webui@file\" URL: \"http://192.168.0.110:17010\" Weight: 1 Reason: received error status code: 401","time":"2019-10-07T16:34:42+02:00"}
{"level":"error","msg":"server not found","time":"2019-10-07T16:34:42+02:00"}
{"level":"warning","msg":"Health check failed: Remove from server list. Backend: \"homepage@file\" URL: \"http://192.168.0.110:80\" Weight: 1 Reason: received error status code: 404","time":"2019-10-07T16:34:42+02:00"}
{"level":"error","msg":"server not found","time":"2019-10-07T16:34:42+02:00"}
{"level":"warning","msg":"Health check failed: Remove from server list. Backend: \"nas-nas1-webui@file\" URL: \"https://192.168.0.110:5001\" Weight: 1 Reason: received error status code: 404","time":"2019-10-07T16:34:42+02:00"}
{"level":"warning","msg":"Health check failed: Remove from server list. Backend: \"nas-nas1-webui@file\" URL: \"https://192.168.0.110:5001\" Weight: 1 Reason: received error status code: 404","time":"2019-10-07T16:34:42+02:00"}
{"level":"error","msg":"server not found","time":"2019-10-07T16:34:42+02:00"}
{"level":"debug","msg":"Stopping current health check goroutines of backend: nas-nas1-webui@file","time":"2019-10-07T16:34:42+02:00"}
{"level":"debug","msg":"Provider connection established with docker 19.03.2 (API 1.40)","providerName":"docker","time":"2019-10-07T16:34:42+02:00"}
{"container":"traefik-traefik-3acc759a84fcd2ffff4fa8f9f83d22288da4d586b255d49ff6088611435e0888","level":"debug","msg":"Filtering disabled container","providerName":"docker","time":"2019-10-07T16:34:42+02:00"}
{"level":"debug","msg":"Configuration received from provider docker: {\"http\":{},\"tcp\":{}}","providerName":"docker","time":"2019-10-07T16:34:42+02:00"}
{"level":"debug","msg":"Refreshing health check for backend: service1-webui@file","time":"2019-10-07T16:34:52+02:00"}
{"level":"debug","msg":"Refreshing health check for backend: homepage@file","time":"2019-10-07T16:34:52+02:00"}
{"level":"warning","msg":"Health check still failing. Backend: \"service1-webui@file\" URL: \"http://192.168.0.110:17010\" Reason: received error status code: 401","time":"2019-10-07T16:34:52+02:00"}
{"level":"warning","msg":"Health check still failing. Backend: \"homepage@file\" URL: \"http://192.168.0.110:80\" Reason: received error status code: 404","time":"2019-10-07T16:34:52+02:00"}
{"level":"debug","msg":"Refreshing health check for backend: nas-nas1-webui@file","time":"2019-10-07T16:34:52+02:00"}
{"level":"warning","msg":"Health check still failing. Backend: \"nas-nas1-webui@file\" URL: \"https://192.168.0.110:5001\" Reason: received error status code: 404","time":"2019-10-07T16:34:52+02:00"}
{"entryPointName":"http","level":"debug","middlewareName":"pipelining","middlewareType":"Pipelining","msg":"Creating middleware","routerName":"nas-nas1-insecure@file","serviceName":"nas-nas1-webui","time":"2019-10-07T16:34:58+02:00"}
{"entryPointName":"http","level":"debug","msg":"Creating load-balancer","routerName":"nas-nas1-insecure@file","serviceName":"nas-nas1-webui","time":"2019-10-07T16:34:58+02:00"}
{"entryPointName":"http","level":"debug","msg":"Creating server 0 https://192.168.0.110:5001","routerName":"nas-nas1-insecure@file","serverName":0,"serviceName":"nas-nas1-webui","time":"2019-10-07T16:34:58+02:00"}
{"entryPointName":"http","level":"debug","middlewareName":"tracing","middlewareType":"TracingForwarder","msg":"Added outgoing tracing middleware nas-nas1-webui","routerName":"nas-nas1-insecure@file","time":"2019-10-07T16:34:58+02:00"}
{"entryPointName":"http","level":"debug","middlewareName":"pipelining","middlewareType":"Pipelining","msg":"Creating middleware","routerName":"service1-insecure@file","serviceName":"service1-webui","time":"2019-10-07T16:34:58+02:00"}
{"entryPointName":"http","level":"debug","msg":"Creating load-balancer","routerName":"service1-insecure@file","serviceName":"service1-webui","time":"2019-10-07T16:34:58+02:00"}
{"entryPointName":"http","level":"debug","msg":"Creating server 0 http://192.168.0.110:17010","routerName":"service1-insecure@file","serverName":0,"serviceName":"service1-webui","time":"2019-10-07T16:34:58+02:00"}
{"entryPointName":"http","level":"debug","middlewareName":"tracing","middlewareType":"TracingForwarder","msg":"Added outgoing tracing middleware service1-webui","routerName":"service1-insecure@file","time":"2019-10-07T16:34:58+02:00"}
{"entryPointName":"http","level":"debug","middlewareName":"tracing","middlewareType":"TracingForwarder","msg":"Added outgoing tracing middleware api@internal","routerName":"traefik-insecure@file","time":"2019-10-07T16:34:58+02:00"}
{"entryPointName":"http","level":"debug","middlewareName":"pipelining","middlewareType":"Pipelining","msg":"Creating middleware","routerName":"home-insecure@file","serviceName":"homepage","time":"2019-10-07T16:34:58+02:00"}
{"entryPointName":"http","level":"debug","msg":"Creating load-balancer","routerName":"home-insecure@file","serviceName":"homepage","time":"2019-10-07T16:34:58+02:00"}
{"entryPointName":"http","level":"debug","msg":"Creating server 0 http://192.168.0.110:80","routerName":"home-insecure@file","serverName":0,"serviceName":"homepage","time":"2019-10-07T16:34:58+02:00"}
{"entryPointName":"http","level":"debug","middlewareName":"tracing","middlewareType":"TracingForwarder","msg":"Added outgoing tracing middleware homepage","routerName":"home-insecure@file","time":"2019-10-07T16:34:58+02:00"}
{"entryPointName":"http","level":"debug","middlewareName":"traefik-internal-recovery","middlewareType":"Recovery","msg":"Creating middleware","time":"2019-10-07T16:34:58+02:00"}
{"level":"debug","msg":"Setting up healthcheck for service nas-nas1-webui@file with [Hostname:  Headers: map[] Path: /_health Port: 0 Interval: 10s Timeout: 3s]","serviceName":"nas-nas1-webui@file","time":"2019-10-07T16:34:58+02:00"}
{"level":"debug","msg":"Setting up healthcheck for service service1-webui@file with [Hostname:  Headers: map[] Path: /_health Port: 0 Interval: 10s Timeout: 3s]","serviceName":"service1-webui@file","time":"2019-10-07T16:34:58+02:00"}
{"level":"debug","msg":"Setting up healthcheck for service homepage@file with [Hostname:  Headers: map[] Path: /_health Port: 0 Interval: 10s Timeout: 3s]","serviceName":"homepage@file","time":"2019-10-07T16:34:58+02:00"}
{"level":"debug","msg":"Setting up healthcheck for service nas-nas1-webui@file with [Hostname:  Headers: map[] Path: /_health Port: 0 Interval: 10s Timeout: 3s]","serviceName":"nas-nas1-webui@file","time":"2019-10-07T16:34:58+02:00"}
{"level":"debug","msg":"Setting up healthcheck for service service1-webui@file with [Hostname:  Headers: map[] Path: /_health Port: 0 Interval: 10s Timeout: 3s]","serviceName":"service1-webui@file","time":"2019-10-07T16:34:58+02:00"}
{"level":"debug","msg":"Setting up healthcheck for service homepage@file with [Hostname:  Headers: map[] Path: /_health Port: 0 Interval: 10s Timeout: 3s]","serviceName":"homepage@file","time":"2019-10-07T16:34:58+02:00"}
{"level":"debug","msg":"No default certificate, generating one","time":"2019-10-07T16:34:58+02:00"}
{"level":"debug","msg":"Initial health check for backend: \"homepage@file\"","time":"2019-10-07T16:34:58+02:00"}
{"level":"debug","msg":"Stopping current health check goroutines of backend: service1-webui@file","time":"2019-10-07T16:34:58+02:00"}
{"level":"debug","msg":"Initial health check for backend: \"service1-webui@file\"","time":"2019-10-07T16:34:58+02:00"}
{"level":"debug","msg":"Stopping current health check goroutines of backend: nas-nas1-webui@file","time":"2019-10-07T16:34:58+02:00"}
{"level":"debug","msg":"Stopping current health check goroutines of backend: homepage@file","time":"2019-10-07T16:34:58+02:00"}
{"level":"debug","msg":"Initial health check for backend: \"nas-nas1-webui@file\"","time":"2019-10-07T16:34:58+02:00"}
{"level":"debug","msg":"Initial health check for backend: \"service1-webui@file\"","time":"2019-10-07T16:34:58+02:00"}
{"level":"debug","msg":"Initial health check for backend: \"homepage@file\"","time":"2019-10-07T16:34:58+02:00"}
{"level":"warning","msg":"Health check failed: Remove from server list. Backend: \"homepage@file\" URL: \"http://192.168.0.110:80\" Weight: 1 Reason: received error status code: 404","time":"2019-10-07T16:34:58+02:00"}
{"level":"debug","msg":"Initial health check for backend: \"nas-nas1-webui@file\"","time":"2019-10-07T16:34:58+02:00"}
{"level":"warning","msg":"Health check failed: Remove from server list. Backend: \"service1-webui@file\" URL: \"http://192.168.0.110:17010\" Weight: 1 Reason: received error status code: 401","time":"2019-10-07T16:34:58+02:00"}
{"level":"warning","msg":"Health check failed: Remove from server list. Backend: \"service1-webui@file\" URL: \"http://192.168.0.110:17010\" Weight: 1 Reason: received error status code: 401","time":"2019-10-07T16:34:58+02:00"}
{"level":"error","msg":"server not found","time":"2019-10-07T16:34:58+02:00"}
{"level":"debug","msg":"Stopping current health check goroutines of backend: service1-webui@file","time":"2019-10-07T16:34:58+02:00"}
{"level":"warning","msg":"Health check failed: Remove from server list. Backend: \"nas-nas1-webui@file\" URL: \"https://192.168.0.110:5001\" Weight: 1 Reason: received error status code: 404","time":"2019-10-07T16:34:58+02:00"}
{"level":"debug","msg":"Stopping current health check goroutines of backend: nas-nas1-webui@file","time":"2019-10-07T16:34:58+02:00"}
{"level":"warning","msg":"Health check failed: Remove from server list. Backend: \"homepage@file\" URL: \"http://192.168.0.110:80\" Weight: 1 Reason: received error status code: 404","time":"2019-10-07T16:34:58+02:00"}
{"level":"error","msg":"server not found","time":"2019-10-07T16:34:58+02:00"}
{"level":"debug","msg":"Stopping current health check goroutines of backend: homepage@file","time":"2019-10-07T16:34:58+02:00"}
{"level":"warning","msg":"Health check failed: Remove from server list. Backend: \"nas-nas1-webui@file\" URL: \"https://192.168.0.110:5001\" Weight: 1 Reason: received error status code: 404","time":"2019-10-07T16:34:58+02:00"}
{"level":"error","msg":"server not found","time":"2019-10-07T16:34:58+02:00"}
{"level":"debug","msg":"Refreshing health check for backend: homepage@file","time":"2019-10-07T16:35:08+02:00"}
{"level":"debug","msg":"Refreshing health check for backend: service1-webui@file","time":"2019-10-07T16:35:08+02:00"}
{"level":"warning","msg":"Health check still failing. Backend: \"homepage@file\" URL: \"http://192.168.0.110:80\" Reason: received error status code: 404","time":"2019-10-07T16:35:08+02:00"}
{"level":"warning","msg":"Health check still failing. Backend: \"service1-webui@file\" URL: \"http://192.168.0.110:17010\" Reason: received error status code: 401","time":"2019-10-07T16:35:08+02:00"}
{"level":"debug","msg":"Refreshing health check for backend: nas-nas1-webui@file","time":"2019-10-07T16:35:08+02:00"}
{"level":"warning","msg":"Health check still failing. Backend: \"nas-nas1-webui@file\" URL: \"https://192.168.0.110:5001\" Reason: received error status code: 404","time":"2019-10-07T16:35:08+02:00"}

PROBLEMS

  • I can CURL NAS services from everywhere in my LAN (e.g.: curl -s http://192.168.0.110:80)
  • I also can access to these services in my favorite browser (e.g.: http://192.168.0.110:80)
  • BUT Traefik can't proxify them (HTTP 404)... Or if he can, it seems that it is stuck because of, maybe, invalid headers? Because I get HTTP 401 on some services.
  • AND when I tried to enable Traefik Dashboard, it only worked with the setting "insecure: true". I was not able to make it work with a route (e.g.: Host(`admin.domain.fr`) && PathPrefix(`/proxy`) ).

Any help would be greatly appreciated :star_struck: .

2

One of your issues is that your Traefik config is only set up for http traffic (port 80). It does not see any of the https requests because you don't have port 443 exposed, nor is 443 an entrypoint. In the logs you see this message repeated

..."No default certificate, generating one"...

Traefik is trying to find/create a certificate, and this is where it's stuck. You need to define a certificate resolver and a location to store your cert.

I would also suggest keeping it simple and going service-by-service. Start by getting the Traefik dashboard to display, then add the other routers, middlewares, and services to your setup from there. This way you can more easily identify where the problem is.

1 Like