How to debug "Bad gateway" response?

Traefik Traefik v3 (latest)
,
1

I've got Traefik running in docker with a service defined in a configuration file. The specific service was always problematic to begin with - it's Microsoft DevOps Server and Traefik seems to not handle the Windows authentication well. But the issues I've faced in the past were related to the authentication, not general availability.

I wanted to check if the new Traefik release changes anything. Now I'm getting a Bad gateway response from Traefik for this service and nothing else. Nothing in the logs (even with the debug level).

If I set up a local hosts file mapping the service IP to the host name defined in the file I'm able to access the service (i.e. the service works fine and responds by the expected hostname internally).

Since Traefik's working inside Docker I've opened a console and checked if I can ping the target service and there was a connection. Besides, Traefik's panel shows the service status as "success". So there's no connection issue I can see between Traefik and the target service.

Any idea how to "debug" such "Bad gateway" issues?

2

Bad gateway means Traefik can't reach the target service.

This usually happens with providers.docker, when the target serive is connected to multiple Docker networks, not all networks overlap with Traefik and providers.docker.network is not set.

It happens because Traefik will pick a random Docker network IP of the target service to connect, but it might be an IP from a Docker network it can't actually reach. So simply tell Traefik which Docker network to use.

Note that the network name needs to be the one Docker is using. If you use compose, the network name might change as it is usually prefixed with the compose project name. So either use an external created network, use name: in compose to set the network name or use the (prefixed) project network name.

3

@bluepuma77
The target service is not a docker-based service. And it's not the only service we have that's file-defined and off-docker and the other services work fine. Your suggestions would be great if this was something entirely docker-based, but sadly, it's not (only Traefik is in this case).

The service exists in our internal network and its name is resolvable by our internal DNS which I've confirmed by pinging the service name from an SH console opened in Traefik's working docker container (DNS name got resolved => confirms correct name + IP got a response => confirms connectivity).

4

Traefik container image includes wget (not curl), try to connect to the target service.

5

@bluepuma77
I did, but wget throws an SSL error. AFAIK, Traefik should ignore such issues via a insecureSkipVerify: true transport setting. The target service, using the configured hostname, is using an invalid certificate (because it's currently a test case anyway).

Here's the config file for reference.

http:
 routers:
   to-devops:
     rule: "Host(`devops-test.domain.com`)"
     service: devops
     entrypoints: websecure
     tls:
       certResolver: le
 services:
   devops:
     loadBalancer:
       serversTransport: devops-transport
       servers:
       - url: https://srv-tfs-p03.domain.local:443
 serversTransports:
   devops-transport:
     disableHTTP2: true
     insecureSkipVerify: true
6 
wget --no-check-certificate https://example.com
7

That helped.

I think I know what the issue is / was. It seems like Traefik isn't passing the host header...? When I configured the target service to accept srv-tfs-p03.domain.local as a valid domain name then it started to work.

Which is weird, because, as stated, if I set up a HOSTS file I can use the devops-test.domain.com just fine, but Traefik couldn't use it.

8

Check passHostHeader doc.

9

passHostHeader is true by default and I wasn't setting it anywhere in the configuration. Also the Traefik panel shows the Pass host header flag as set.

image
image483×556 11 KB