Traefik is loading forever - Gateway Timeout on dotnet core api

Hi there,

i trying to migrate from nginx to traefik. So far it works fine, except my own backend (dotnet core web api). This is loading about 2 minutes and then i get gateway timeout error. In the logs from the traefik container i dont even see and traffic coming in for this service / route (api).

This is how my nginx was configured (at least the relevant part):

server {
       server_name api.blablabla.de;

       location / {
   	proxy_set_header      X-Real-IP         $remote_addr;
               proxy_set_header      X-Forwarded-For   $proxy_add_x_forwarded_for;
               proxy_set_header      X-Forwarded-Proto https;
               proxy_set_header      X-Forwarded-Port  443;
               proxy_set_header      Host              $host;
   	proxy_request_buffering off;
               proxy_read_timeout 180;
               proxy_connect_timeout 180;
               proxy_redirect  off;
               proxy_pass http://localhost:50000;
       }


   location /images/ {
   	add_header Access-Control-Allow-Origin *;
       	proxy_pass http://localhost:50000;
   }

       listen 443 ssl;
   ssl_certificate /etc/letsencrypt/live/marcogriep.de-0001/fullchain.pem; # managed by Certbot
   ssl_certificate_key /etc/letsencrypt/live/marcogriep.de-0001/privkey.pem; # managed by Certbot
}

This is my traefik docker-compose (i removed unrelevant services by ....)

version: '3.7'

services:
  traefik:
    # The latest official supported Traefik docker image
    image: traefik:v2.5.3
    # Enables the Traefik Dashboard and tells Traefik to listen to docker
    # enable --log.level=INFO so we can see what Traefik is doing in the log files
    ports:
      # Exposes port 80 for incomming web requests
      - "80:80"
      - "443:443"
      # The Web UI port http://0.0.0.0:8080 (enabled by --api.insecure=true)
      - "8080:8080"
    volumes:
      # So that Traefik can listen to the Docker events
      - /var/run/docker.sock:/var/run/docker.sock
       # Mounts the Traefik static configuration inside the Traefik container
      - ./traefik.yml:/etc/traefik/traefik.yml

  prometheus:
    ...
  
  grafana:
    ...

  website:
    ...


  # Error Page service
  error:
    ...

  db:
    image: mariadb
    environment:
      MYSQL_ROOT_PASSWORD: password
      MYSQL_DATABASE: api_db
      MYSQL_USER: dbuser
      MYSQL_PASSWORD: password
    volumes:
      - db_german_outdoors_api_data:/var/lib/mysql

  # General-Backend
  api:
    image: registry.gitlab.com/griepmarco/german-outdoors-packlight-api:latest
    environment:
      APP_KEY: password
      DB_HOST: db
      DB_NAME: api_db
      DB_USER: root
      DB_PASSWORD: password
      DB_PORT: 3306
      MAIL_PROVIDER: smtp
      MAIL_SERVER: mail.localhost
      MAIL_PORT: 587
      MAIL_FROM_NAME: Test
      MAIL_USERNAME: mail@localhost
      MAIL_PASSWORD: password
      MAIL_USESSL: 1
      APP_URL: https://localhost
      IMAGE_STORE_PATH: images
    depends_on:
      - db
    volumes:
      - db_german_outdoors_image_data:/app/wwwroot/images
    deploy:
      labels:
       - "traefik.enable=true"
       - "traefik.http.routers.api.rule=Host(`api.localhost`)"
       - "traefik.http.routers.api.service=api"
       - "traefik.http.routers.api.entrypoints=web"
       - "traefik.http.services.api.loadbalancer.server.port=80"
       - "traefik.http.middlewares.testheader.headers.accesscontrolallowmethods=GET,PUT,OPTIONS,DELETE,POST"
       - "traefik.http.middlewares.testheader.headers.accesscontrolalloworigin=*"
       - "traefik.http.middlewares.testheader.headers.accessControlAllowCredentials=true"
       - "traefik.http.middlewares.testheader.headers.isDevelopment=true"
       - "traefik.http.middlewares.testheader.headers.accesscontrolmaxage=100"
      restart_policy:
        condition: on-failure

  db_users:
      ...

  keycloak:
      ...

volumes:
    db_german_outdoors_api_data: {}
    db_german_outdoors_image_data: {}
    prometheus_data: {}
    grafana_data: {}
    users_data: {}

All other routes and services are working as expected. The API is up and did all the database migrations and is waiting for requests. In the API logs i dont see any requests either... Using Traefik in Docker Swarm Mode. All those CORS Middleware was just a test if this could solve my issue. I also removed the overlay network. Same issues.

Traefik logs

time="2021-10-23T09:17:47Z" level=error msg="service \"traefik-db-users\" error: port is missing" container=traefik-db-users-nbkdokmupz9580pgr4i3t0bhp providerName=docker
time="2021-10-23T09:17:47Z" level=error msg="field not found, node: accesscontrolalloworigin" providerName=docker container=traefik-api-gc04t5efiomvb7yzobioq5gkb
time="2021-10-23T09:17:47Z" level=error msg="service \"traefik-db\" error: port is missing" providerName=docker container=traefik-db-rya99gfw001fft24ydx23yfvh
time="2021-10-23T09:17:47Z" level=error msg="service \"traefik-traefik\" error: port is missing" providerName=docker container=traefik-traefik-sbe6p0hxp4pthlbfb2idmy5uz
time="2021-10-23T09:17:47Z" level=debug msg="Configuration received from provider docker: {\"http\":{\"routers\":{\"catapp\":{\"entryPoints\":[\"web\"],\"service\":\"catapp\",\"rule\":\"Host(`catapp.localhost`)\"},\"error\":{\"entryPoints\":[\"web\"],\"service\":\"error\",\"rule\":\"Host(`error.localhost`)\"},\"grafana\":{\"entryPoints\":[\"web\"],\"service\":\"grafana\",\"rule\":\"Host(`metrics.localhost`)\"},\"keycloak\":{\"entryPoints\":[\"web\"],\"service\":\"keycloak\",\"rule\":\"Host(`sso.localhost`)\"},\"prometheus\":{\"entryPoints\":[\"web\"],\"service\":\"prometheus\",\"rule\":\"Host(`prometheus.localhost`)\"},\"website\":{\"entryPoints\":[\"web\"],\"middlewares\":[\"test-compress\",\"test-errorpages\"],\"service\":\"website\",\"rule\":\"HostRegexp(`{host:(www.)?localhost}`)\"}},\"services\":{\"catapp\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.4.3:5000\"}],\"passHostHeader\":true}},\"error\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.4.18:80\"}],\"passHostHeader\":true}},\"grafana\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.4.16:3000\"}],\"passHostHeader\":true}},\"keycloak\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.4.20:8080\"}],\"passHostHeader\":true}},\"prometheus\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.4.22:9090\"}],\"passHostHeader\":true}},\"website\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.4.12:80\"}],\"passHostHeader\":true}}},\"middlewares\":{\"test-compress\":{\"compress\":{}},\"test-errorpages\":{\"errors\":{\"status\":[\"400-599\"],\"service\":\"error\",\"query\":\"/{status}.html\"}}}},\"tcp\":{},\"udp\":{}}" providerName=docker
time="2021-10-23T09:17:47Z" level=info msg="Skipping same configuration" providerName=docker
10.0.4.22 - - [23/Oct/2021:09:17:49 +0000] "GET /metrics HTTP/1.1" 200 2721 "-" "-" 375 "prometheus@internal" "-" 8ms
10.0.4.22 - - [23/Oct/2021:09:17:54 +0000] "GET /metrics HTTP/1.1" 200 2724 "-" "-" 376 "prometheus@internal" "-" 6ms
10.0.4.22 - - [23/Oct/2021:09:17:59 +0000] "GET /metrics HTTP/1.1" 200 2722 "-" "-" 377 "prometheus@internal" "-" 4ms
time="2021-10-23T09:18:02Z" level=error msg="service \"traefik-db-users\" error: port is missing" container=traefik-db-users-nbkdokmupz9580pgr4i3t0bhp providerName=docker
time="2021-10-23T09:18:02Z" level=error msg="field not found, node: accesscontrolalloworigin" providerName=docker container=traefik-api-gc04t5efiomvb7yzobioq5gkb
time="2021-10-23T09:18:02Z" level=error msg="service \"traefik-db\" error: port is missing" providerName=docker container=traefik-db-rya99gfw001fft24ydx23yfvh
time="2021-10-23T09:18:02Z" level=error msg="service \"traefik-traefik\" error: port is missing" providerName=docker container=traefik-traefik-sbe6p0hxp4pthlbfb2idmy5uz
time="2021-10-23T09:18:02Z" level=debug msg="Configuration received from provider docker: {\"http\":{\"routers\":{\"catapp\":{\"entryPoints\":[\"web\"],\"service\":\"catapp\",\"rule\":\"Host(`catapp.localhost`)\"},\"error\":{\"entryPoints\":[\"web\"],\"service\":\"error\",\"rule\":\"Host(`error.localhost`)\"},\"grafana\":{\"entryPoints\":[\"web\"],\"service\":\"grafana\",\"rule\":\"Host(`metrics.localhost`)\"},\"keycloak\":{\"entryPoints\":[\"web\"],\"service\":\"keycloak\",\"rule\":\"Host(`sso.localhost`)\"},\"prometheus\":{\"entryPoints\":[\"web\"],\"service\":\"prometheus\",\"rule\":\"Host(`prometheus.localhost`)\"},\"website\":{\"entryPoints\":[\"web\"],\"middlewares\":[\"test-compress\",\"test-errorpages\"],\"service\":\"website\",\"rule\":\"HostRegexp(`{host:(www.)?localhost}`)\"}},\"services\":{\"catapp\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.4.3:5000\"}],\"passHostHeader\":true}},\"error\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.4.18:80\"}],\"passHostHeader\":true}},\"grafana\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.4.16:3000\"}],\"passHostHeader\":true}},\"keycloak\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.4.20:8080\"}],\"passHostHeader\":true}},\"prometheus\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.4.22:9090\"}],\"passHostHeader\":true}},\"website\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.4.12:80\"}],\"passHostHeader\":true}}},\"middlewares\":{\"test-compress\":{\"compress\":{}},\"test-errorpages\":{\"errors\":{\"status\":[\"400-599\"],\"service\":\"error\",\"query\":\"/{status}.html\"}}}},\"tcp\":{},\"udp\":{}}" providerName=docker
time="2021-10-23T09:18:02Z" level=info msg="Skipping same configuration" providerName=docker
10.0.4.22 - - [23/Oct/2021:09:18:04 +0000] "GET /metrics HTTP/1.1" 200 2731 "-" "-" 378 "prometheus@internal" "-" 2ms

All Routes and Services are without errors or warnings in the dashboard.

How can i troubleshoot whats the problem? Any other logs?

@MarcoGriep88 Hi, did you figure out how to resolve it?

Could you please try to update the port of your backend application? I noticed that in example you showed you are using the port 50000.

traefik.http.services.<service_name>.loadbalancer.server.port - the port configured by that label refers to the container port, see the attached docs below:

1 Like