[SOLVED] Using Traefik before Varnish with proxyProtocol

[ribba]

Hi,

Trying to replace HAProxy with Traefik v2.4 in front of Varnish.
HAProxy is doing TLS termination and forwarding traffic to Varnish using proxy protocol v2.

HAProxy config
backend varnish server varnish 127.0.0.1:6083 send-proxy-v2

Trying to replicate this in Traefik v2.4 without luck.
I guess there is something wrong with the route config. Been trying different things with the router rules with no luck.
Also tried it inside http instead of tcp.
Traefik is running inside docker with network hosts. This explains localhost vs 127.0.0.1.

dynamic configuration
`
tcp:
services:
varnish:
loadBalancer:
proxyProtocol:
version: 2
servers:
- address: "localhost:6083"

routers:
varnish-router:
service: "varnish"
tls:
domains:
- main: "example.com"
`

For the most, Traefik ends in 404 page not found
Any ideas? This should be strait forward shouldn't...

[jakubhajek]

Hello @ribba

Thanks for using Traefik.

Seems your configuration is missing Rule directive to route incoming requests to the specific service.

Would you please try to test it?

[ribba]

Hi,

thanks for answering @jakubhajek

I've tried these rules with no luck:
rule: "HostSNI(example.com)"
rule: "Host(example.com)"
rule: "PathPrefix(/)"

One thing I wonder, should the router be placed inside tcp or http ?
As the browser talks http with Traefik it make sense to put the router inside http, and the backend inside tcp as Traefik talks to Varnish using proxy protocol (tcp).

[jakubhajek]

Hello @ribba

Would you please try to attach that configuration:

tcp:
  routers:
    Router-1:
      entryPoints:
        - web
      rule: "HostSNI(`*`)"
      service: "varnish-svc"

  services:
    varnish-svc:
      loadBalancer:
        servers:
          - address: "varnish:6081"

It worked on my lab but my test environment was pretty poor. However, I was able to get 200 by sending a request directly to Traefik, then Traefik forwarded the request to Varnish and finally it hit my test application running behind Varnish.

Please let me know what are the results of your tests.

Thank you,

[ribba]

Thanks @jakubhajek. Got an expected respons from Varnish now.
However, next issue: As soon as tcp is added to the mix things break apart again. tls: {} is enough.

tcp:
  services:
    varnish:
      loadBalancer:
        proxyProtocol:
          version: 2
        servers:
        - address: "varnish:8443"

  routers:
    varnish-router:
      rule: "HostSNI(`*`)"
      service: "varnish"
      tls: {}

I have removed my http router completely, but my tls used to be like this:

      tls:
        certResolver: "sample"
        domains:
          - main: "example.com"

My resolver is simple:

certificatesResolvers:
  sample:
    acme:
      email: "bill@example.com"
      storage: "/etc/traefik/acme/acme-letsencrypt.json"
      httpChallenge:
        entryPoint: "http"

Doing curl to port 443:

$ curl -k --header "Host: example.com" https://127.0.0.1
curl: (56) Unexpected EOF

From the log:

time="2021-05-03T20:18:08Z" level=debug msg="Handling connection from 172.17.0.1:46788"
time="2021-05-03T20:18:08Z" level=debug msg="Serving default certificate for request: \"\""
time="2021-05-03T20:18:08Z" level=debug msg="Error while terminating connection: close tcp 172.17.0.4:52042->172.17.0.3:8443: shutdown: transport endpoint is not connected"
time="2021-05-03T20:18:08Z" level=error msg="Error during connection: readfrom tcp 172.17.0.4:52042->172.17.0.3:8443: write tcp 172.17.0.4:52042->172.17.0.3:8443: write: broken pipe"

Doing curl to port 80

$ curl -k --header "Host: example.com" http://127.0.0.1
404 page not found

with nothing in the log.

Running Varnish 6.0.7. Even had to build my own image for xkey and proxy protocol support.

[jakubhajek]

Hello @ribba

Thanks a lot for providing the details of your configuration.
From my side, I can say that I've spent some time trying to reproduce the issue and provide you a working solution. My stack is following:

incoming rq -> Traefik -> Varnish -> a web server with a static page.

Can you please try to use that config:

tcp:
  routers:
    Router-1:
      entryPoints:
        - websecure
      rule: "HostSNI(`*`)"
      service: "varnish-svc"
      tls: {}

  services:
    varnish-svc:
      loadBalancer:
        servers:
          - address: "varnish:80"

I removed by purpose the proxy arguments and it works. Would you please try the config I suggested, you should get a valid response from the backend - in my case, it was just a simple web server with a static web page.

Please let me know,
Thank you,

[ribba]

I appreciate your effort on this

This last setup is the same as having a normal http router, which talks http with Varnish. This is working, but it not how things should be done. With this setup Traefik needs to pass information like client ip, http protocol used (http or https) etc via headers to Varnish.

It is what the proxy protocol solves.

See all the header_up statements used in this Caddy example you need to add to the mix so that the application behind Varnish can make correct redirections etc.

Reverse proxy with Caddy 2

The fact that proxy protocol towards Varnish is working until tls is added, could it be a bug or missing feature in Traefik's handling of this?

HAProxy towards Varnish over proxy protocol is dead simple. Just use Varnish's provided port for proxy protocol (usually 6083 or 8443) and add send-proxy-v2.

backend varnish server varnish 127.0.0.1:6083 send-proxy-v2

btw, Caddy 2 does not (yet) support talking proxy protocol to backends

[jakubhajek]

Hello @ribba

I understand this completely. Thanks again for clarifying that.

I did further investigation and the below config should work correctly.

tcp:
  routers:
    Router-1:
      entryPoints:
        - websecure
      rule: "HostSNI(`*`)"
      service: "varnish-svc"
      tls: {}

  services:
    varnish-svc:
      loadBalancer:
        servers:
          - address: "varnish:80"
        proxyProtocol:
          version: 2

By default, Varnish accepts HTTP requests. We force Traefik to send requests to service using the PROXY protocol we need to update the Varnish configuration to accepst that communication. In order to do that we have to add the keyword PROXY on the port where Varnish is listening to.

So the arguments passed to the Varnish has to contain the following line:

"-a :80,PROXY"

and then Varnish will be able to accept Proxy.

I hope that helps.

Thank you, Jakub

[ribba]

Indeed!
Think the problem is http/https protocol confusion. me, Traefik, or both
Not mixing entrypoints in tcp routes seems to make a difference. This works like a charm in my test bench:

tcp:
  services:
    varnish:
      loadBalancer:
        proxyProtocol:
          version: 2
        servers:
        - address: "varnish:8443"

  routers:
    varnish-router-http:
      rule: "HostSNI(`*`)"
      service: "varnish"
      entrypoints:
        - "http"

    varnish-router-https:
      rule: "HostSNI(`*`)"
      service: "varnish"
      entrypoints:
        - "https"
      tls: {}

[ribba]

Hi @jakubhajek

Nailed it, finally ! Problem was http/2 all along.

Traefik is telling the world I support http/2, but my Varnish was not.
Forcing curl into http1.1 revealed the problem:
curl --http1.1 -v https://example.com worked !

After enabling http/2 in Varnish this finally worked.
The backend of Varnish (Apache 2.4) does not need to have http/2 enabled.
Hope this thread find someone else...

Thanks for your patience @jakubhajek.

[jakubhajek]

Hello @ribba

Thanks for your patience and for your willingness to use Traefik.

It was a good conversation, thanks for sharing all details concerning your implementation. Yesterday I started to create a repo with that use case, so I will update it later on by adding extra configuration.

Stay tuned Jakub