PassTLSClientCert not populating X-Forwarded-Tls-Client-Cert

Traefik Traefik v2
,
1

I have 2 instances of an application running in separate VMs, vm1 & vm2.
The app is containerised, each instance consists of a traefik container & a rails app (t1 & r1 run in vm1, t2 & r2 run in vm2).
There are also db containers (d1 & d2) but that's irrelevant to the question.

t1 & t2 are configured to run with ssl, certificates are based on a self-signed CA.
r1 & r2 are running without ssl - port 80.

All this setup is working as expected, however there is also an rpc mechanism, where r1 creates a Net::HTTP request with a client certificate, which is dispatched directly to t2, which should verify the client cert against it's CA (available to traefik). RPC requests always have a url path '/rpc/call', and a separate traefik router is configured to managed the client cert handling, which is only needed for rpc.

t2 forwards he request to r2. r2 needs to perform some additional validation, for which it requires access to the requesters client cert. From the traefik documentation I see the PassTLSClientCert option which is intended just for this purpose - to forward the client certificate in the request to r2.

Client certs are also based on a self-signed CA.

What I'm seeing is that traefik is creating the X-Forwarded-Tls-Client-Cert header, but it's empty and I'm unable to figure out what is wrong with my config.

Is the header not forwarded because the app is not https ?

Traefik static conf:

global:
  checkNewVersion: false
  sendAnonymousUsage: false
entryPoints:
  web:
    address: ":80"
  web-secure:
    address: ":443"
log:
  level: "DEBUG"
api:
  insecure: true
providers:
  file:
    filename: "/etc/traefik/conf/traefik-dynamic.yml"
    watch: true
  docker:
    exposedByDefault: false
    network: "traefik_proxy"

Dynamic:

http:
  middlewares:
    redirect-to-https:
       redirectScheme:
         scheme: https
    passtlsclientcert:
      passTLSClientCert:
        pem: true
        info:
          sans: true
tls:
  stores:
    default:
       defaultCertificate:
         certFile: /etc/traefik/certs/server.crt
         keyFile:  /etc/traefik/certs/server.key
  certificates:
  options:
    default:
      minVersion: VersionTLS12
    app-rpc-options:
      clientAuth:
        caFiles:
          - /etc/traefik/certs/clientAppCA.pem
        clientAuthType: RequireAndVerifyClientCert```

traefik labels in app docker-compose:

    labels:
      # traefik-v2:
      - "traefik.enable=true"

      # Forward http to https:
      - "traefik.http.routers.${HOSTNAME}-app-http.rule=Host(`${HOSTNAME}.domain.loc`)"
      - "traefik.http.routers.${HOSTNAME}-app-http.entryPoints=web"
      - "traefik.http.routers.${HOSTNAME}-app-http.middlewares=redirect-to-https@file"

      # https:
      - "traefik.http.routers.${HOSTNAME}-app-https.rule=Host(`${HOSTNAME}.domain.loc`)"
      - "traefik.http.routers.${HOSTNAME}-app-https.entryPoints=web-secure"
      - "traefik.http.routers.${HOSTNAME}-app-https.tls=true"

      # Incoming RPC calls:
      - "traefik.http.routers.app-rpc.middlewares=passtlsclientcert@file"
      - "traefik.http.routers.app-rpc.rule=Host(`${HOSTNAME}.domain.loc/rpc/call`)"
      - "traefik.http.routers.app-rpc.entryPoints=web-secure"
      - "traefik.http.routers.app-rpc.tls=true"
      # This is allowed: https://doc.traefik.io/traefik/https/tls
      - "traefik.http.routers.app-rpc.tls.options=app-rpc-options@file"

Logs:

Start traefik:

time="2022-03-27T16:51:36Z" level=info msg="Configuration loaded from file: /etc/traefik/conf/traefik-static.yml"
time="2022-03-27T16:51:36Z" level=info msg="Traefik version 2.6.1 built on 2022-02-14T16:50:25Z"
time="2022-03-27T16:51:36Z" level=debug msg="Static configuration loaded {\"global\":{},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"traefik\":{\"address\":\":8080\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"udp\":{\"timeout\":\"3s\"}},\"web\":{\"address\":\":80\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"udp\":{\"timeout\":\"3s\"}},\"web-secure\":{\"address\":\":443\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"udp\":{\"timeout\":\"3s\"}}},\"providers\":{\"providersThrottleDuration\":\"2s\",\"docker\":{\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"network\":\"traefik_proxy\",\"swarmModeRefreshSeconds\":\"15s\"},\"file\":{\"watch\":true,\"filename\":\"/etc/traefik/conf/traefik-dynamic.yml\"}},\"api\":{\"insecure\":true,\"dashboard\":true},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"},\"pilot\":{\"dashboard\":true}}"
time="2022-03-27T16:51:36Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
time="2022-03-27T16:51:36Z" level=info msg="Starting provider aggregator.ProviderAggregator"
time="2022-03-27T16:51:36Z" level=debug msg="Start TCP Server" entryPointName=web
time="2022-03-27T16:51:36Z" level=info msg="Starting provider *file.Provider"
time="2022-03-27T16:51:36Z" level=debug msg="*file.Provider provider configuration: {\"watch\":true,\"filename\":\"/etc/traefik/conf/traefik-dynamic.yml\"}"
time="2022-03-27T16:51:36Z" level=info msg="Starting provider *traefik.Provider"
time="2022-03-27T16:51:36Z" level=debug msg="*traefik.Provider provider configuration: {}"
time="2022-03-27T16:51:36Z" level=info msg="Starting provider *acme.ChallengeTLSALPN"
time="2022-03-27T16:51:36Z" level=debug msg="*acme.ChallengeTLSALPN provider configuration: {\"Timeout\":4000000000}"
time="2022-03-27T16:51:36Z" level=debug msg="Start TCP Server" entryPointName=traefik
time="2022-03-27T16:51:36Z" level=debug msg="Start TCP Server" entryPointName=web-secure
time="2022-03-27T16:51:36Z" level=debug msg="Configuration received from provider file: {\"http\":{\"middlewares\":{\"passtlsclientcert\":{\"passTLSClientCert\":{\"pem\":true,\"info\":{\"sans\":true}}},\"redirect-to-https\":{\"redirectScheme\":{\"scheme\":\"https\"}}}},\"tcp\":{},\"udp\":{},\"tls\":{\"options\":{\"app-rpc-options\":{\"clientAuth\":{\"clientAuthType\":\"RequireAndVerifyClientCert\"},\"alpnProtocols\":[\"h2\",\"http/1.1\",\"acme-tls/1\"]},\"default\":{\"minVersion\":\"VersionTLS12\",\"clientAuth\":{},\"alpnProtocols\":[\"h2\",\"http/1.1\",\"acme-tls/1\"]}},\"stores\":{\"default\":{}}}}" providerName=file
time="2022-03-27T16:51:36Z" level=debug msg="Configuration received from provider internal: {\"http\":{\"routers\":{\"api\":{\"entryPoints\":[\"traefik\"],\"service\":\"api@internal\",\"rule\":\"PathPrefix(`/api`)\",\"priority\":2147483646},\"dashboard\":{\"entryPoints\":[\"traefik\"],\"middlewares\":[\"dashboard_redirect@internal\",\"dashboard_stripprefix@internal\"],\"service\":\"dashboard@internal\",\"rule\":\"PathPrefix(`/`)\",\"priority\":2147483645}},\"services\":{\"api\":{},\"dashboard\":{},\"noop\":{}},\"middlewares\":{\"dashboard_redirect\":{\"redirectRegex\":{\"regex\":\"^(http:\\\\/\\\\/(\\\\[[\\\\w:.]+\\\\]|[\\\\w\\\\._-]+)(:\\\\d+)?)\\\\/$\",\"replacement\":\"${1}/dashboard/\",\"permanent\":true}},\"dashboard_stripprefix\":{\"stripPrefix\":{\"prefixes\":[\"/dashboard/\",\"/dashboard\"]}}},\"serversTransports\":{\"default\":{\"maxIdleConnsPerHost\":200}}},\"tcp\":{},\"tls\":{}}" providerName=internal
time="2022-03-27T16:51:36Z" level=debug msg="Added outgoing tracing middleware api@internal" routerName=api@internal middlewareName=tracing middlewareType=TracingForwarder entryPointName=traefik
time="2022-03-27T16:51:36Z" level=debug msg="Added outgoing tracing middleware dashboard@internal" middlewareName=tracing middlewareType=TracingForwarder entryPointName=traefik routerName=dashboard@internal
time="2022-03-27T16:51:36Z" level=debug msg="Creating middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix
time="2022-03-27T16:51:36Z" level=debug msg="Adding tracing to middleware" routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal entryPointName=traefik
time="2022-03-27T16:51:36Z" level=debug msg="Creating middleware" middlewareType=RedirectRegex entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal
time="2022-03-27T16:51:36Z" level=debug msg="Setting up redirection from ^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$ to ${1}/dashboard/" middlewareType=RedirectRegex entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal
time="2022-03-27T16:51:36Z" level=debug msg="Adding tracing to middleware" routerName=dashboard@internal middlewareName=dashboard_redirect@internal entryPointName=traefik
time="2022-03-27T16:51:36Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=traefik middlewareName=traefik-internal-recovery
time="2022-03-27T16:51:36Z" level=info msg="Starting provider *docker.Provider"
time="2022-03-27T16:51:36Z" level=debug msg="*docker.Provider provider configuration: {\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"network\":\"traefik_proxy\",\"swarmModeRefreshSeconds\":\"15s\"}"
time="2022-03-27T16:51:36Z" level=debug msg="Provider connection established with docker 19.03.15 (API 1.40)" providerName=docker
time="2022-03-27T16:51:36Z" level=debug msg="Filtering disabled container" providerName=docker container=reverse-proxy-traefik-e0d0c622d8996de5df6b0a763eaeeb7f1500822c0c0ea9265a197a0fd453e10f
time="2022-03-27T16:51:36Z" level=debug msg="Filtering disabled container" providerName=docker container=db-mariadb-d1ea425cfc5c365a9b7ae221c4455448027e77587ff6c8a917c98e743daeeda0
time="2022-03-27T16:51:36Z" level=debug msg="Configuration received from provider docker: {\"http\":{},\"tcp\":{},\"udp\":{}}" providerName=docker
time="2022-03-27T16:51:36Z" level=debug msg="Added outgoing tracing middleware api@internal" middlewareType=TracingForwarder entryPointName=traefik routerName=api@internal middlewareName=tracing
time="2022-03-27T16:51:36Z" level=debug msg="Added outgoing tracing middleware dashboard@internal" middlewareName=tracing middlewareType=TracingForwarder entryPointName=traefik routerName=dashboard@internal
time="2022-03-27T16:51:36Z" level=debug msg="Creating middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix
time="2022-03-27T16:51:36Z" level=debug msg="Adding tracing to middleware" routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal entryPointName=traefik
time="2022-03-27T16:51:36Z" level=debug msg="Creating middleware" middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex entryPointName=traefik routerName=dashboard@internal
time="2022-03-27T16:51:36Z" level=debug msg="Setting up redirection from ^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$ to ${1}/dashboard/" routerName=dashboard@internal middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex entryPointName=traefik
time="2022-03-27T16:51:36Z" level=debug msg="Adding tracing to middleware" middlewareName=dashboard_redirect@internal entryPointName=traefik routerName=dashboard@internal
time="2022-03-27T16:51:36Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery

Start r2

time="2022-03-27T16:52:39Z" level=debug msg="Provider event received {Status:start ID:6ddeb25c604b4ac9926ceb296a66f35341332bdb07ba4e03b01847827bb39130 From:domain/app:8.0.0alpha18 Type:container Action:start Actor:{ID:6ddeb25c604b4ac9926ceb296a66f35341332bdb07ba4e03b01847827bb39130 Attributes:map[com.docker.compose.config-hash:49d050895b43bb03d5280e1fe59958f620216ae097a8a11af7de8d5850338b71 com.docker.compose.container-number:1 com.docker.compose.oneoff:False com.docker.compose.project:app com.docker.compose.project.config_files:docker-compose.yml com.docker.compose.project.working_dir:/data/apps/app/app/app-8.0.0alpha18 traefik.enable:true traefik.http.routers.app-rpc.entryPoints:web-secure traefik.http.routers.app-rpc.middlewares:passtlsclientcert@file traefik.http.routers.app-rpc.rule:Host(`vm2.domain.loc/rpc/call`) traefik.http.routers.app-rpc.tls:true traefik.http.routers.app-rpc.tls.options:app-rpc-options@file traefik.http.routers.vm2-app-https.entryPoints:web-secure traefik.http.routers.vm2-app-https.rule:Host(`vm2.domain.loc`) traefik.http.routers.vm2-app-https.tls:true]} Scope:local Time:1648399959 TimeNano:1648399959126711644}" providerName=docker
time="2022-03-27T16:52:39Z" level=debug msg="Filtering disabled container" providerName=docker container=reverse-proxy-traefik-e0d0c622d8996de5df6b0a763eaeeb7f1500822c0c0ea9265a197a0fd453e10f
time="2022-03-27T16:52:39Z" level=debug msg="Filtering disabled container" providerName=docker container=db-mariadb-d1ea425cfc5c365a9b7ae221c4455448027e77587ff6c8a917c98e743daeeda0
time="2022-03-27T16:52:39Z" level=debug msg="Configuration received from provider docker: {\"http\":{\"routers\":{\"app-rpc\":{\"entryPoints\":[\"web-secure\"],\"middlewares\":[\"passtlsclientcert@file\"],\"service\":\"rails-app\",\"rule\":\"Host(`vm2.domain.loc/rpc/call`)\",\"tls\":{\"options\":\"app-rpc-options@file\"}},\"vm2-app-https\":{\"entryPoints\":[\"web-secure\"],\"service\":\"rails-app\",\"rule\":\"Host(`vm2.domain.loc`)\",\"tls\":{}}},\"services\":{\"rails-app\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.23.0.3:3000\"}],\"passHostHeader\":true}}}},\"tcp\":{},\"udp\":{}}" providerName=docker
time="2022-03-27T16:52:39Z" level=debug msg="Added outgoing tracing middleware dashboard@internal" routerName=dashboard@internal middlewareName=tracing middlewareType=TracingForwarder entryPointName=traefik
time="2022-03-27T16:52:39Z" level=debug msg="Creating middleware" middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix entryPointName=traefik routerName=dashboard@internal
time="2022-03-27T16:52:39Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_stripprefix@internal
time="2022-03-27T16:52:39Z" level=debug msg="Creating middleware" middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex entryPointName=traefik routerName=dashboard@internal
time="2022-03-27T16:52:39Z" level=debug msg="Setting up redirection from ^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$ to ${1}/dashboard/" middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex entryPointName=traefik routerName=dashboard@internal
time="2022-03-27T16:52:39Z" level=debug msg="Adding tracing to middleware" entryPointName=traefik routerName=dashboard@internal middlewareName=dashboard_redirect@internal
time="2022-03-27T16:52:39Z" level=debug msg="Added outgoing tracing middleware api@internal" middlewareName=tracing middlewareType=TracingForwarder entryPointName=traefik routerName=api@internal
time="2022-03-27T16:52:39Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=traefik middlewareName=traefik-internal-recovery
time="2022-03-27T16:52:39Z" level=debug msg="Creating middleware" serviceName=rails-app middlewareName=pipelining middlewareType=Pipelining entryPointName=web-secure routerName=app-rpc@docker
time="2022-03-27T16:52:39Z" level=debug msg="Creating load-balancer" entryPointName=web-secure routerName=app-rpc@docker serviceName=rails-app
time="2022-03-27T16:52:39Z" level=debug msg="Creating server 0 http://172.23.0.3:3000" routerName=app-rpc@docker serviceName=rails-app serverName=0 entryPointName=web-secure
time="2022-03-27T16:52:39Z" level=debug msg="child http://172.23.0.3:3000 now UP"
time="2022-03-27T16:52:39Z" level=debug msg="Propagating new UP status"
time="2022-03-27T16:52:39Z" level=debug msg="Added outgoing tracing middleware rails-app" routerName=app-rpc@docker middlewareName=tracing middlewareType=TracingForwarder entryPointName=web-secure
time="2022-03-27T16:52:39Z" level=debug msg="Creating middleware" middlewareType=PassClientTLSCert entryPointName=web-secure routerName=app-rpc@docker middlewareName=passtlsclientcert@file
time="2022-03-27T16:52:39Z" level=debug msg="Adding tracing to middleware" entryPointName=web-secure routerName=app-rpc@docker middlewareName=passtlsclientcert@file
time="2022-03-27T16:52:39Z" level=debug msg="Creating middleware" middlewareType=Pipelining entryPointName=web-secure routerName=vm2-app-https@docker serviceName=rails-app middlewareName=pipelining
time="2022-03-27T16:52:39Z" level=debug msg="Creating load-balancer" entryPointName=web-secure routerName=vm2-app-https@docker serviceName=rails-app
time="2022-03-27T16:52:39Z" level=debug msg="Creating server 0 http://172.23.0.3:3000" serverName=0 entryPointName=web-secure routerName=vm2-app-https@docker serviceName=rails-app
time="2022-03-27T16:52:39Z" level=debug msg="child http://172.23.0.3:3000 now UP"
time="2022-03-27T16:52:39Z" level=debug msg="Propagating new UP status"
time="2022-03-27T16:52:39Z" level=debug msg="Added outgoing tracing middleware rails-app" entryPointName=web-secure routerName=vm2-app-https@docker middlewareName=tracing middlewareType=TracingForwarder
time="2022-03-27T16:52:39Z" level=debug msg="Creating middleware" entryPointName=web-secure middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2022-03-27T16:52:39Z" level=debug msg="Adding route for vm2.domain.loc/rpc/call with TLS options app-rpc-options@file" entryPointName=web-secure
time="2022-03-27T16:52:39Z" level=debug msg="Adding route for vm2.domain.loc with TLS options default" entryPointName=web-secure

RPC call produceds:

time="2022-03-27T16:53:22Z" level=debug msg="Serving default certificate for request: \"vm2.domain.loc\""
time="2022-03-27T16:53:22Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/rpc/call\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip;q=1.0,deflate;q=0.6,identity;q=0.3\"],\"Connection\":[\"close\"],\"Content-Length\":[\"107\"],\"Content-Type\":[\"application/x-www-form-urlencoded\"],\"User-Agent\":[\"Ruby\"],\"X-Forwarded-Host\":[\"vm2.domain.loc\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik\"],\"X-Real-Ip\":[\"172.16.11.181\"]},\"ContentLength\":107,\"TransferEncoding\":null,\"Host\":\"vm2.domain.loc\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"rpcAddr\":\"172.16.11.181:35944\",\"RequestURI\":\"/rpc/call\",\"TLS\":null}"
time="2022-03-27T16:53:22Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/rpc/call\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip;q=1.0,deflate;q=0.6,identity;q=0.3\"],\"Connection\":[\"close\"],\"Content-Length\":[\"107\"],\"Content-Type\":[\"application/x-www-form-urlencoded\"],\"User-Agent\":[\"Ruby\"],\"X-Forwarded-Host\":[\"vm2.domain.loc\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik\"],\"X-Real-Ip\":[\"172.16.11.181\"]},\"ContentLength\":107,\"TransferEncoding\":null,\"Host\":\"vm2.domain.loc\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"rpcAddr\":\"172.16.11.181:35944\",\"RequestURI\":\"/rpc/call\",\"TLS\":null}" ForwardURL="http://172.23.0.3:3000"
time="2022-03-27T16:53:22Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/rpc/call\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip;q=1.0,deflate;q=0.6,identity;q=0.3\"],\"Connection\":[\"close\"],\"Content-Length\":[\"107\"],\"Content-Type\":[\"application/x-www-form-urlencoded\"],\"User-Agent\":[\"Ruby\"],\"X-Forwarded-Host\":[\"vm2.domain.loc\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik\"],\"X-Real-Ip\":[\"172.16.11.181\"]},\"ContentLength\":107,\"TransferEncoding\":null,\"Host\":\"vm2.domain.loc\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"rpcAddr\":\"172.16.11.181:35944\",\"RequestURI\":\"/rpc/call\",\"TLS\":null}"