TCP proxyprotocol setup fails

We recently decided to switch to Traefik v2 because we require proxy protocol support for TCP based services we run behind a Linode loadbalancer and a Traefik instance. As far as I understand, this feature has been added with release v2.4.

I managed to setup a new Traefik instance and get everything running without the loadbalancer but as soon as I enabled the loadbalancer and consequently the proxy protocol headers, I get

dial tcp connect: connection refused

Traefik logs:

"Error while Peeking first byte: proxyproto: unsupported address family and protocol"

I could not find anything useful about this specific error. This is my traefik configuration:

    level = "DEBUG"
        address = ":80"

          insecure = true
      dashboard = true
      insecure = true


The loadbalancer is configured to use TCP and ProxyProtocol v1. I also tested v2 and it threw the same error.

Any help would be appreciated.