As I am trying to debug why a traefik load balancer doesn't work (router does not appear in dashboard UI, routes 404), I wondered how I could get more insights into what is happening. I do not get any errors anywhere (logs, UI), but I found a screenshot via Google that showed errors where I have never seen them before. Do I miss some static debug setup to enable this, or is this from some feature that never got implemented?
The scenario I am currently in:
# static.yml
debug: true
log:
level: DEBUG
filePath: /var/log/traefik/error.log
format: json
accessLog:
filePath: /var/log/traefik/access.log
format: json # common
# (i) Performance: Increase log lines to buffer before writing to file.
bufferingSize: 1
The whole stack is built upon Docker Compose, so most conf happens via the Docker Provider a.k.a. labels. Only tls conf and Middleware use the File Provider via a dynamic.yml file. The reference is set in Docker labels by calling it with the @file suffix.
The following Nodejs container is not showing up in the dashboard and non-routable.
# docker-compose.nodejs.yml
---
version: '3'
services:
nodejs:
image: "node:${NODEJS_TAG:-current-alpine}"
build:
context: ${PWD}/nodejs
# Only accessible during build time
args:
NODE_PORT: ${NODE_PORT:-3001}
NODE_ENV: ${NODE_ENV:-production}
NPM_CONFIG_PREFIX: ${NODE_CONFIG_PREFIX:-/home/node/.npm-global}
NODE_EXTRA_CA_CERTS: "/usr/local/share/ca-certificates/rootCA.pem"
SSL_CERT_DIR: "/usr/local/share/ca-certificates"
hostname: "nodejs"
command: sh ./bin/start ./app/server.js # <<<< Here's the spdy ExpressJS server, listening on Port 3001
restart: on-failure
security_opt:
- no-new-privileges:true
networks:
- default
expose:
- ${NODE_PORT:-3001}
env_file:
- ${PWD}/admin/dist/.env
environment:
NODE_PORT: ${NODE_PORT:-3001}
NODE_ENV: ${NODE_ENV}
NPM_CONFIG_PREFIX: ${NODE_CONFIG_PREFIX:-/home/node/.npm-global}
working_dir: /usr/src/data
volumes:
- ${PWD}/admin/dist:/usr/src/data
- cert-storage:/usr/local/share/ca-certificates
- nodejs-hidden-modules:/usr/src/data/node_modules # (i) The `node_modules` volume
labels:
- "traefik.enable=true"
- "traefik.docker.network=${DEFAULT_NETWORK}"
- "traefik.http.routers.yolo-router.rule=Host(`yolo.${ROOT_DOMAIN}`)"
- "traefik.http.routers.yolo-router.entrypoints=https"
- "traefik.http.routers.yolo-router.tls=true"
- "traefik.http.routers.yolo-router.tls.options=default"
- "traefik.http.routers.yolo-router.service=yolo"
- "traefik.http.routers.yolo-router.middlewares=secured@file"
- "traefik.http.services.yolo.loadbalancer.server.port=3001"
The exposed port is 3001 and I can reach the container from other containers without a problem:
$ docker-compose exec notyolo /bin/sh
wget -qO- nodejs:3001
# prints "yolo"
This is everything I get in the logs:
==> ./logs/traefik/error.log <==
{"level":"debug","msg":"Serving default certificate for request: \"yolo.demo.test\"","time":"2021-01-29T23:24:49+01:00"}
==> ./logs/traefik/access.log <==
{"ClientAddr":"172.21.0.1:61322","ClientHost":"172.21.0.1","ClientPort":"61322","ClientUsername":"-","DownstreamContentSize":19,"DownstreamStatus":404,"Duration":1082400,"OriginContentSize":19,"OriginDuration":116700,"OriginStatus":404,
"Overhead":965700,"RequestAddr":"yolo.demo.test","RequestContentSize":0,"RequestCount":13418,"RequestHost":"yolo.demo.test","RequestMethod":"GET","RequestPath":"/","RequestPort":"-","RequestProtocol":"HTTP/2.0","RequestScheme":"https","
RetryAttempts":0,"StartLocal":"2021-01-29T23:24:49.8978335+01:00","StartUTC":"2021-01-29T22:24:49.8978335Z","entryPointName":"https","level":"info","msg":"","time":"2021-01-29T23:24:49+01:00"}
Calling the container via Traefik from the external/ host/ default network, results in a 404:
$ curl -k -L -v https://yolo.demo.test
* Trying 127.0.0.1:443...
* Connected to yolo.bell.test (127.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /Users/user/Library/Application Support/mkcert/rootCA.pem
* CApath: /Users/user/Library/Application Support/mkcert/rootCA.pem
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: O=mkcert development certificate; OU=user@yolo.local (Firstname Lastname)
* start date: Jan 24 23:02:26 2021 GMT
* expire date: Apr 24 22:02:26 2023 GMT
* issuer: O=mkcert development CA; OU=user@yolo.local (Firstname Lastname); CN=mkcert user@yolo.local (Firstname Lastname)
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7faa26012000)
> GET / HTTP/2
> Host: yolo.demo.test
> user-agent: curl/7.74.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 404
< content-type: text/plain; charset=utf-8
< x-content-type-options: nosniff
< content-length: 19
< date: Fri, 29 Jan 2021 22:24:49 GMT
<
404 page not found
* Connection #0 to host yolo.demo.test left intact
Any idea how I can continue debugging what's wrong with the load balancer config?