HomeAssistant SmartThings not working as I can't GET api/webhook working with Cloudflare as a proxy

[xabix]

Hello Experts,

I have reading and trying several thinks but I am unable to get api/webhook to my HomeAssistant.

Here is the json access.log output:

{"ClientAddr":"172.71.127.103:35708","ClientHost":"myPUBLICip","ClientPort":"35708","ClientUsername":"-","DownstreamContentSize":0,"DownstreamStatus":405,"Duration":1781616,"OriginContentSize":0,"OriginDuration":1436007,"OriginStatus":0,"Overhead":345609,"RequestAddr":"ha.mySITE.com","RequestContentSize":0,"RequestCount":1074,"RequestHost":"ha.mySITE.com","RequestMethod":"GET","RequestPath":"/api/webhook/5b6a7a8dc1519e5b8d04a8eee5284271cfd06e8fd5a73600XXXXXXX42f6","RequestPort":"-","RequestProtocol":"HTTP/2.0","RequestScheme":"https","RetryAttempts":0,"RouterName":"ha-rtr@file","ServiceAddr":"192.168.10.50:8123","ServiceName":"ha-svc@file","ServiceURL":"http://192.168.10.50:8123","StartLocal":"2024-09-04T09:03:46.249780195+02:00","StartUTC":"2024-09-04T07:03:46.249780195Z","TLSCipher":"TLS_AES_128_GCM_SHA256","TLSVersion":"1.3","entryPointName":"https","level":"info","msg":"","time":"2024-09-04T09:03:46+02:00"}

In traefik.log I get:

2024-09-03T17:52:41+02:00 DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:301 > Creating server entryPointName=https routerName=ha-rtr@file serverName=4e4c6da362850e96 serviceName=ha-svc@file target=http://192.168.10.50:8123
2024-09-03T17:52:41+02:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/chain/chain.go:22 > Creating middleware entryPointName=https middlewareName=chain-no-auth@file middlewareType=Chain routerName=ha-rtr@file
2024-09-03T17:52:41+02:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/compress/compress.go:39 > Creating middleware entryPointName=https middlewareName=middlewares-compress@file middlewareType=Compress routerName=ha-rtr@file
2024-09-03T17:52:41+02:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=https middlewareName=middlewares-compress@file routerName=ha-rtr@file
2024-09-03T17:52:41+02:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/headers/headers.go:28 > Creating middleware entryPointName=https middlewareName=middlewares-secure-headers@file middlewareType=Headers routerName=ha-rtr@file
2024-09-03T17:52:41+02:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/headers/headers.go:42 > Setting up secureHeaders from {map[] map[X-Robots-Tag:none,noarchive,nosnippet,notranslate,noimageindex, server:] false [] [GET OPTIONS PUT] [] [] [] 100 false [] [X-Forwarded-Host] map[] 63072000 true true true false allow-from https:mySITE.com true true    same-origin camera=(), microphone=(), geolocation=(), payment=(), usb=(), vr=() false <nil> <nil> <nil> <nil> <nil>} entryPointName=https middlewareName=middlewares-secure-headers@file middlewareType=Headers routerName=ha-rtr@file
2024-09-03T17:52:41+02:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/headers/headers.go:48 > Setting up customHeaders/Cors from {map[] map[X-Robots-Tag:none,noarchive,nosnippet,notranslate,noimageindex, server:] false [] [GET OPTIONS PUT] [] [] [] 100 false [] [X-Forwarded-Host] map[] 63072000 true true true false allow-from https:mySITE.com true true    same-origin camera=(), microphone=(), geolocation=(), payment=(), usb=(), vr=() false <nil> <nil> <nil> <nil> <nil>} entryPointName=https middlewareName=middlewares-secure-headers@file middlewareType=Headers routerName=ha-rtr@file
2024-09-03T17:52:41+02:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=https middlewareName=middlewares-secure-headers@file routerName=ha-rtr@file
2024-09-03T17:52:41+02:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:29 > Creating middleware entryPointName=https middlewareName=middlewares-https-redirectscheme@file middlewareType=RedirectScheme routerName=ha-rtr@file
2024-09-03T17:52:41+02:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:30 > Setting up redirection to https  entryPointName=https middlewareName=middlewares-https-redirectscheme@file middlewareType=RedirectScheme routerName=ha-rtr@file
2024-09-03T17:52:41+02:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/ratelimiter/rate_limiter.go:49 > Creating middleware entryPointName=https middlewareName=middlewares-rate-limit@file middlewareType=RateLimiter routerName=ha-rtr@file
2024-09-03T17:52:41+02:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/extractor.go:45 > Using IPStrategy entryPointName=https middlewareName=middlewares-rate-limit@file middlewareType=RateLimiter routerName=ha-rtr@file
2024-09-03T17:52:41+02:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=https middlewareName=middlewares-rate-limit@file routerName=ha-rtr@file
....
2024-09-04T09:03:46+02:00 DBG github.com/traefik/traefik/v3/pkg/server/service/loadbalancer/wrr/wrr.go:196 > Service selected by WRR: 4e4c6da362850e96
2024-09-04T09:04:02+02:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/compress/compress.go:103 > Unable to parse MIME type error="mime: no media type" middlewareName=middlewares-compress@file middlewareType=Com
press
2024-09-04T09:04:02+02:00 DBG github.com/traefik/traefik/v3/pkg/middlewares/compress/compress.go:103 > Unable to parse MIME type error="mime: no media type" middlewareName=middlewares-compress@file middlewareType=Com
press

In Cloudflare I do proxy the DNS resolution on my ha.xxx.com host

In my rule I have:

http:
  routers:
    ha-rtr:
      rule: "Host(`ha.{{env "DOMAINNAME_CLOUD_SERVER"}}`)"
      entryPoints:
        - https
      middlewares:
        - chain-no-auth@file
      priority: 100
      service: ha-svc
      tls: {}
  services:
    ha-svc:
      loadBalancer:
        servers:
          - url: "http://192.168.10.50:8123"

I tried to add another router like:

    ha-webhook-rtr:
      rule: "Host(`ha.{{env "DOMAINNAME_CLOUD_SERVER"}}`) && ( PathPrefix(`/api/websocket`) )"
      entryPoints:
        - https
      middlewares:
        - chain-no-auth@file
      priority: 110
      service: ha-svc
      tls: {}

Any idea of the issue? and what I am missing ?

In another thread, it seems that in 2021 they disabled TLSv1.3 but I don't like trying such idea as I feel this is not what should be done nowadays

Thank you so much in advance!!
Mr X

[bluepuma77]

I don’t get it, what do you want to achieve with this config? Why have two routers with the same config going to the same service? The second router will make no difference.

Do you have other Traefik middlewares enabled (compress, headers, rate-limiter)? Try to disable them.

[xabix]

Bonjour bluepuma77,

In order for some HA pluggns to work like SmartThings, my HA instance does need to respond to a webhook query coming from SmarThings servers.

I have only one router at the moment towards my ha (but I tried two as initially the host had authentication through Google Auth but I wanted to remove such for webhook queries).

At the moment I have those middlewares (and routers/services)

My current chain and middlewares are:

    chain-no-auth:
      chain:
        middlewares:
          - middlewares-rate-limit
          - middlewares-https-redirectscheme
          - middlewares-secure-headers
          - middlewares-compress

    middlewares-rate-limit:
      rateLimit:
        average: 200
        burst: 100

    middlewares-https-redirectscheme:
      redirectScheme:
        scheme: https
        permanent: true

   middlewares-secure-headers:
      headers:
        accessControlAllowMethods:
          - GET
          - OPTIONS
          - PUT
        accessControlMaxAge: 100
        hostsProxyHeaders:
          - "X-Forwarded-Host"
        stsSeconds: 63072000
        stsIncludeSubdomains: true
        stsPreload: true
        forceSTSHeader: true
        customFrameOptionsValue: "allow-from https:{{env "DOMAINNAME_CLOUD_SERVER"}}" #
        contentTypeNosniff: true
        browserXssFilter: true
        referrerPolicy: "same-origin"
        permissionsPolicy: "camera=(), microphone=(), geolocation=(), payment=(), usb=(), vr=()"
        customResponseHeaders:
          X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex,"
          server: ""
          # https://community.traefik.io/t/how-to-make-websockets-work-with-traefik-2-0-setting-up-rancher/1732
          # X-Forwarded-Proto: "https"

    middlewares-compress:
      compress: {}

which is working for quite some services but maybe they were changes needed to be done in v3 or some could be enhanced

Thanks for the suggestion, I will do specif middleware-chain for HomeAssistant and try remove one by one some middlewares

Anything that raise a bell in the current middlewares-secure-headers maybe?
Not sure what type of accessMethods could be this webhook query.

Merci

[bluepuma77]

OriginStatus=0 indicates that the request never arrived at the target service. compress shows an error, but it is only classified as DBG. I would try to disable headers and compress.

In general HA is just a single instance, so you should only need one router wit a sub-domain for it.

[xabix]

@xabix

        forceSTSHeader: true

When you say disble headers you mean to put this to false?

I did

    chain-no-auth4ha:
      chain:
        middlewares:
          - middlewares-rate-limit
          - middlewares-https-redirectscheme
          - middlewares-secure-headers4ha
          #- middlewares-compress

I duplicated the existing middlewares-secure-headers to middlewares-secure-headers4ha and put to false forceSTSHeader

No better results (meaning with no compress in the chain and forceSTSHeader to false).

Trying this gives me a 404:

{"ClientAddr":"172.71.122.174:52684","ClientHost":"myIP","ClientPort":"52684","ClientUsername":"-","DownstreamContentSize":19,"DownstreamStatus":404,"Duration":113370,"GzipRatio":0,"OriginContentSize":0,"OriginDuration":0,"OriginStatus":0,"Overhead":113370,"RequestAddr":"ha.mySITE.com","RequestContentSize":0,"RequestCount":1401,"RequestHost":"ha.mySITE.com","RequestMethod":"GET","RequestPath":"/api/websocket","RequestPort":"-","RequestProtocol":"HTTP/1.1","RequestScheme":"https","RetryAttempts":0,"StartLocal":"2024-09-04T11:31:21.405296808+02:00","StartUTC":"2024-09-04T09:31:21.405296808Z","TLSCipher":"TLS_AES_128_GCM_SHA256","TLSVersion":"1.3","entryPointName":"https","level":"info","msg":"","time":"2024-09-04T11:31:21+02:00"}
{"ClientAddr":"172.71.119.47:15470","ClientHost":"myIP","ClientPort":"15470","ClientUsername":"-","DownstreamContentSize":19,"DownstreamStatus":404,"Duration":92960,"GzipRatio":0,"OriginContentSize":0,"OriginDuration":0,"OriginStatus":0,"Overhead":92960,"RequestAddr":"ha.mySITE.com","RequestContentSize":0,"RequestCount":1402,"RequestHost":"ha.mySITE.com","RequestMethod":"GET","RequestPath":"/api/websocket","RequestPort":"-","RequestProtocol":"HTTP/1.1","RequestScheme":"https","RetryAttempts":0,"StartLocal":"2024-09-04T11:31:22.508520592+02:00","StartUTC":"2024-09-04T09:31:22.508520592Z","TLSCipher":"TLS_AES_128_GCM_SHA256","TLSVersion":"1.3","entryPointName":"https","level":"info","msg":"","time":"2024-09-04T11:31:22+02:00"}
{"ClientAddr":"172.71.131.100:35768","ClientHost":"myIP","ClientPort":"35768","ClientUsername":"-","DownstreamContentSize":19,"DownstreamStatus":404,"Duration":109490,"GzipRatio":0,"OriginContentSize":0,"OriginDuration":0,"OriginStatus":0,"Overhead":109490,"RequestAddr":"ha.mySITE.com","RequestContentSize":0,"RequestCount":1403,"RequestHost":"ha.mySITE.com","RequestMethod":"POST","RequestPath":"/api/config/config_entries/flow/01J6Y598W1ZPXK3YJQBBVPSRxx","RequestPort":"-","RequestProtocol":"HTTP/2.0","RequestScheme":"https","RetryAttempts":0,"StartLocal":"2024-09-04T11:31:22.876046078+02:00","StartUTC":"2024-09-04T09:31:22.876046078Z","TLSCipher":"TLS_AES_128_GCM_SHA256","TLSVersion":"1.3","entryPointName":"https","level":"info","msg":"","time":"2024-09-04T11:31:22+02:00"}
{"ClientAddr":"172.69.222.72:55482","ClientHost":"myIP","ClientPort":"55482","ClientUsername":"-","DownstreamContentSize":19,"DownstreamStatus":404,"Duration":75880,"GzipRatio":0,"OriginContentSize":0,"OriginDuration":0,"OriginStatus":0,"Overhead":75880,"RequestAddr":"ha.mySITE.com","RequestContentSize":0,"RequestCount":1404,"RequestHost":"ha.mySITE.com","RequestMethod":"GET","RequestPath":"/api/websocket","RequestPort":"-","RequestProtocol":"HTTP/1.1","RequestScheme":"https","RetryAttempts":0,"StartLocal":"2024-09-04T11:31:24.609309375+02:00","StartUTC":"2024-09-04T09:31:24.609309375Z","TLSCipher":"TLS_AES_128_GCM_SHA256","TLSVersion":"1.3","entryPointName":"https","level":"info","msg":"","time":"2024-09-04T11:31:24+02:00"}
{"ClientAddr":"172.69.223.199:17732","ClientHost":"myIP","ClientPort":"17732","ClientUsername":"-","DownstreamContentSize":19,"DownstreamStatus":404,"Duration":81469,"GzipRatio":0,"OriginContentSize":0,"OriginDuration":0,"OriginStatus":0,"Overhead":81469,"RequestAddr":"ha.mySITE.com","RequestContentSize":0,"RequestCount":1405,"RequestHost":"ha.mySITE.com","RequestMethod":"GET","RequestPath":"/api/websocket","RequestPort":"-","RequestProtocol":"HTTP/1.1","RequestScheme":"https","RetryAttempts":0,"StartLocal":"2024-09-04T11:31:27.71455351+02:00","StartUTC":"2024-09-04T09:31:27.71455351Z","TLSCipher":"TLS_AES_128_GCM_SHA256","TLSVersion":"1.3","entryPointName":"https","level":"info","msg":"","time":"2024-09-04T11:31:27+02:00"}

Could it be that POST needs to be added within accessControlAllowMethods?

[bluepuma77]

Remove the whole middlewares-secure-headers4ha to test.

[xabix]

I just did (keeping middlewares-rate-limit & middlewares-https-redirectscheme)

I don't see much change and we get a 405:

{"ClientAddr":"172.69.223.150:52944","ClientHost":"myIP","ClientPort":"52944","ClientUsername":"-","DownstreamContentSize":0,"DownstreamStatus":405,"Duration":1626296,"OriginContentSize":0,"OriginDuration":1404297,"OriginStatus":405,"Overhead":221999,"RequestAddr":"ha.mySITE.com","RequestContentSize":0,"RequestCount":5311,"RequestHost":"ha.mySITE.com","RequestMethod":"GET","RequestPath":"/api/webhook/16211727e84e09fb25309bc8XXX19335a9d8952096ca03cd3","RequestPort":"-","RequestProtocol":"HTTP/2.0","RequestScheme":"https","RetryAttempts":0,"RouterName":"ha-rtr@file","ServiceAddr":"192.168.10.50:8123","ServiceName":"ha-svc@file","ServiceURL":"http://192.168.10.50:8123","StartLocal":"2024-09-04T15:20:00.52449356+02:00","StartUTC":"2024-09-04T13:20:00.52449356Z","TLSCipher":"TLS_AES_128_GCM_SHA256","TLSVersion":"1.3","entryPointName":"https","level":"info","msg":"","time":"2024-09-04T15:20:00+02:00"}

Not sure to understand where to look into to progress but it seems that Traefik is not forwarding the request to ha, is this correct?

what could differ between a login tcp connexion vs a webhook that would require some tweaking within Traefik?

Merci

PS: Not sure if this helps but here is a this that seem to refer to similar issue link

[xabix]

Home Assisatant do refere to TLSv1.3 not being supported:

Some reverse proxy configuration settings can interfere with communication from SmartThings. For example, TLSv1.3 is not supported. Setting the supported cipher suite too restrictly will prevent handshaking. The following NGINX SSL configuration is known to work

in https://www.home-assistant.io/integrations/smartthings/#setup

My tls is setup like:

tls:
  options:
    tls-opts:
      minVersion: VersionTLS12
    # mintls13:
    #   minVersion: VersionTLS13
      cipherSuites:
        - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
        - TLS_AES_128_GCM_SHA256
        - TLS_AES_256_GCM_SHA384
        - TLS_CHACHA20_POLY1305_SHA256
        - TLS_FALLBACK_SCSV # Client is doing version fallback. See RFC 7507
      curvePreferences:
        - CurveP521
        - CurveP384
      sniStrict: true

So maybe I need to force TLSv1.2 within the middleware?

ChatGPT proposed this tls which I ll try to set only for my ha service:

tls:
  options:
    tls-ha:
      minVersion: VersionTLS12
      cipherSuites:
        - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
        - TLS_AES_128_GCM_SHA256
        - TLS_AES_256_GCM_SHA384
        - TLS_CHACHA20_POLY1305_SHA256
        - TLS_FALLBACK_SCSV
      curvePreferences:
        - CurveP384
      sniStrict: true
      # dhParams:
      #   file: "/path/to/dhparam.pem"
      preferServerCipherSuites: true

I did comment dhParams for now.

And from HA side I can see:

I did the testing proposed:

I still believe that there is something wrong with my Traefik config

Merci

[bluepuma77]

I am a bit lost. You have a HA client and a HA server? Where is TLS? I can’t really believe that a software in 2024 would not support TLS v1.3.

[xabix]

Hi @bluepuma77 , I feel then less lost

Here is my setup:
Internet -> Cloudflare -> OPNsens 1to1 NAT -> Traefik -> HA server

The SmartThings is a internet cloud based solution that sits on the internet and needs to reach my HA instance. When I configure the plugin in HA, he says:

then I add my API key and then the SmartThings server initiate a webhook call towards my ha.mySITE.com site.

it's HA site that does comment about this TLSv1.2:

thus why I tried to initiate this curl query from outside my network to troubleshoot. From your pov, the issue is not in Traefik? Now we get a 405, but I am not sure it's Traefik answering, or if it's HA. Could it be that something is changed that the inital query? I can try to grab a packet capture on the OPNsense if this makes sense.

Merci

PS: Could it be that locally in access in http?

and if i try locally in https i get: