Use external loadbalancer service as https

# Dynamic configuration
http:

  routers:
    website:
      service: website-service
      rule: "Host(`example.com`)"
      tls:
        certResolver: letsencrypt
        entrypoints: websecure

  services:
     website-service:
       loadBalancer:
         servers:
           - url: "https://website.server1.example.com:443"

Using https for the server url I always get a 404. I read that changing the scheme to https fixes this. you can specify a scheme in docker labels:
- "traefik.http.services.service01.loadbalancer.server.scheme=https"
Source
Unfortunately you can't specify the server url using docker labels, you can only do that using a file provider. Is their anyway for me to specify a scheme and a url?