Bad Request But Traefik Container Can Access Service

wstrater · January 15, 2022, 9:09pm

I am running Traefik 2.5.6 on Docker 19.03.02. I have Traefik proxying many containers all configured with labels that I can access through HTTPS (Let's Encrypt) but I have one, homeassistant, that is consistently failing with a 400 Bad Request.

What is frustrating is that I can see the request to homeassistant in the Traefik logs with a 400 status code but when run a shell in the Traefik container, I can curl the same URL successfully. The only middleware I have is an HTTP to HTTPS redirect but I am trying the HTTPS endpoint directly.

When I try to access Home Assistant from my desktop, I get a 400 following redirects and not following redirects.

curl -L -o /dev/null -w "\nStatus: %{http_code}\n"  https://homeassistant.example.com
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    16  100    16    0     0    219      0 --:--:-- --:--:-- --:--:--   219

Status: 400

I check the Traefik logs at debug level and see the following entries.

time="2022-01-15T14:52:59-05:00" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"*/*\"],\"User-Agent\":[\"curl/7.64.1\"],\"X-Forwarded-Host\":[\"homeassistant.example.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"d1a28518adcc\"],\"X-Real-Ip\":[\"192.168.1.110\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"homeassistant.example.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.1.110:52736\",\"RequestURI\":\"/\",\"TLS\":null}"
time="2022-01-15T14:52:59-05:00" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" ForwardURL="http://172.18.0.25:8123" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"*/*\"],\"User-Agent\":[\"curl/7.64.1\"],\"X-Forwarded-Host\":[\"homeassistant.example.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"d1a28518adcc\"],\"X-Real-Ip\":[\"192.168.1.110\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"homeassistant.example.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.1.110:52736\",\"RequestURI\":\"/\",\"TLS\":null}"
time="2022-01-15T14:52:59-05:00" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"*/*\"],\"User-Agent\":[\"curl/7.64.1\"],\"X-Forwarded-Host\":[\"homeassistant.example.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"d1a28518adcc\"],\"X-Real-Ip\":[\"192.168.1.110\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"homeassistant.example.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.1.110:52736\",\"RequestURI\":\"/\",\"TLS\":null}"
192.168.1.110 - - [15/Jan/2022:19:52:59 +0000] "GET / HTTP/2.0" 400 16 "-" "-" 70 "homeassistant-secure@docker" "http://172.18.0.25:8123" 6ms

I copy the URL from the logs and run a shell in the Traefik container and get a 200 if I follow redirects and a 302 if I don't.

docker exec -it traefik sh
apk --no-cache add curl
curl -L -o /dev/null -w "\nStatus: %{http_code}\n" http://172.18.0.25:8123
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  2822  100  2822    0     0   265k      0 --:--:-- --:--:-- --:--:--  265k

Status: 200

Here are the Traefik labels. I included Portainer since that does work and I can't see how they are different.

docker inspect homeassistant portainer | jq '.[].Config.Labels' | grep traefik
  "traefik.docker.network": "frontend",
  "traefik.enable": "true",
  "traefik.http.middlewares.homeassistant-https-redirect.redirectscheme.scheme": "https",
  "traefik.http.routers.homeassistant-secure.entrypoints": "https",
  "traefik.http.routers.homeassistant-secure.rule": "Host(`homeassistant.example.com`)",
  "traefik.http.routers.homeassistant-secure.service": "homeassistant",
  "traefik.http.routers.homeassistant-secure.tls": "true",
  "traefik.http.routers.homeassistant-secure.tls.certresolver": "cloudflare",
  "traefik.http.routers.homeassistant.entrypoints": "http",
  "traefik.http.routers.homeassistant.middlewares": "homeassistant-https-redirect",
  "traefik.http.routers.homeassistant.rule": "Host(`homeassistant.example.com`)",
  "traefik.http.services.homeassistant.loadbalancer.server.port": "8123"
  "traefik.docker.network": "proxy",
  "traefik.enable": "true",
  "traefik.http.middlewares.portainer-https-redirect.redirectscheme.scheme": "https",
  "traefik.http.routers.portainer-secure.entrypoints": "https",
  "traefik.http.routers.portainer-secure.rule": "Host(`portainer.example.com`)",
  "traefik.http.routers.portainer-secure.service": "portainer",
  "traefik.http.routers.portainer-secure.tls": "true",
  "traefik.http.routers.portainer-secure.tls.certresolver": "cloudflare",
  "traefik.http.routers.portainer.entrypoints": "http",
  "traefik.http.routers.portainer.middlewares": "portainer-https-redirect",
  "traefik.http.routers.portainer.rule": "Host(`portainer.example.com`)",
  "traefik.http.services.portainer.loadbalancer.server.port": "9000"

They are on the same network, frontend.

docker inspect traefik homeassistant portainer | jq '[ .[] | {Name: .Name, NetworkMode: .HostConfig.NetworkMode } ]'
[  {
    "Name": "/traefik",
    "NetworkMode": "frontend"
},  {
    "Name": "/homeassistant",
    "NetworkMode": "frontend"
},  {
    "Name": "/portainer",
    "NetworkMode": "frontend"
} ]

The all have IP address in the same 172.18.0.0/16 subnet.

docker inspect traefik homeassistant portainer | jq '[ .[] | {Name: .Name, IPAddress: .NetworkSettings.Networks.frontend.IPAddress } ]'
[ {
    "Name": "/traefik",
    "IPAddress": "172.18.0.4"
}, {
    "Name": "/homeassistant",
    "IPAddress": "172.18.0.25"
}, {
    "Name": "/portainer",
    "IPAddress": "172.18.0.7"
} ]

Thanks for the help. It must be something simple since I have over 20 containers configured similarly and successfully using Ansible but I have been at this for more than a week and getting nowhere.

Wes.

PuppetMaster2k · January 21, 2022, 7:47am

Hi Wes!

I am experiencing the same issue, as I get HTTP Error 400 from Home Assistant. But maybe I can be of help.

In Home Assistant log I get the following error:

ERROR (MainThread) [homeassistant.components.http.forwarded] A request from a reverse proxy was received from XXX.XXX.XXX.XXX, but your HTTP integration is not set-up for reverse proxies

Therefore I needed to add this configuration to HA-config:

http:
  use_x_forwarded_for: true
  trusted_proxies:
    - 172.18.0.4 # your traefik ip

Could that be the reason of your HTTP Error 400, that you are missing a piece of configuration?

What I ended up with, I have to run HA in network_mode: host AND need to provide traefik with extra_hosts, but I always get 400. If I put HA in the same network as traefik, I get a 404 error. Not sure why this is.

PuppetMaster2k · January 21, 2022, 11:05am

What I have found out so far:

with Home Assistant v21.7 came a breacking change, in which it became necessary to set trusted proxies for reverse proxies. Therefore HA looks into the X-FORWARDED Header of the request.

HA needs

X-FORWARDED-FOR: [list of IPs]
X-FORWARDED-PROTO
X-FORWARDED-HOST

snip

Here is a log entry from traefik, when I try to access HA from my URL

Forwarding this request to URL" ForwardURL="http://172.17.0.1:8123" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/static/fonts/roboto/Roboto-Regular.woff2\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7\"],\"Cache-Control\":[\"no-cache\"],\"Dnt\":[\"1\"],\"Pragma\":[\"no-cache\"],\"Referer\":[\"https://homeassistant.example.com/service_worker.js\"],\"Sec-Fetch-Dest\":[\"empty\"],\"Sec-Fetch-Mode\":[\"cors\"],\"Sec-Fetch-Site\":[\"same-origin\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36\"],\"**X-Forwarded-Host**\":[\"homeassistant.example.com\"],\"X-**Forwarded-Port**\":[\"443\"],\"**X-Forwarded-Proto**\":[\"https\"],\"X-Forwarded-Server\":[\"3d9a1e134e40\"],\"X-Real-Ip\":[\"11.111.111.11\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"homeassistant.example.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"11.111.111.11:60227\",\"RequestURI\":\"/static/fonts/roboto/Roboto-Regular.woff2\",\"TLS\":null}"

Have a look here on what HA does:
https://github.com/home-assistant/core/blob/dev/homeassistant/components/http/forwarded.py

Bascically HA expects the IPs in X-FORWARDED-FOR, and if it does not find them throws 400. For now, I have switched to a HA version before v21.7, just to get it working. But that ist not really a solution, as I am not able to update HA, if necessary.

But I definitely want to stay with traefik because it is very easy to configure and use.

EDIT: Ok, its not the missing IPs. After looking into the code again, and checking the error message, it seems more likely, that "use_x_forwarded_for" flag from HA config is not set correctly. Still I don't get it, because it is set.

wstrater · January 21, 2022, 11:48pm

I had seen similar suggestions but they mentioned ip_ban_enabled: true was important and it still kept failing. Tried your settings with internal Docker IP range and Docker host IP range in the list of trusted_proxies and it is now working. Not sure if I opened it up too much since Traefik and Home Assistant are running on the same Docker network.

The fact that HA was failing behind a proxy implies it was already checking the HTTP headers. Telling it use_x_forwarded_for: true seems redundant unless it was checking a different header but I am not sure why since X-Forwarded-For is an industry standard. Maybe only trusted_proxies is needed.

I kept comparing HA and Portainer behind the Traefik reverse proxy. The logs looked similar except HA was failing while Portainer was working. I was hoping for more details from the Traefik logs.

It seems frustrating that HA is so particular about a reverse proxy but people open their home automation to the Internet directly or through Naba Casa so I guess it is a good thing that they are careful.

Here is what I am using as configured in an Ansible template. traefik_network is a response from the docker_network module.

http:
#  ip_ban_enabled: true
  login_attempts_threshold: 5
  use_x_forwarded_for: true
  trusted_proxies:
    - {{ ansible_default_ipv4.network }}/24 # Local LAN Subnet
    - {{ traefik_network.network.IPAM.Config[0].Subnet }} # Docker Subnet

Thanks for the help.

system · January 24, 2022, 11:49pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Traefik as frontend for Home Assistant gives error 400: Bad Request Traefik v2 docker	2	8322	September 21, 2022
Error 400 - bad request Traefik v3 (latest) docker	21	710	May 23, 2024
Problems with port forwarding Traefik v2 docker	1	1261	October 11, 2022
Routing homeassistant in network_mode host Traefik v2 docker , file	2	1087	June 27, 2023
400 Bad Request The plain HTTP request was sent to HTTPS port Traefik v2 docker	1	3127	March 17, 2023

Bad Request But Traefik Container Can Access Service

Related topics