Can I use File to create reuseable middlewares?

My idea is to either create Chains or Middlewares externally using the providers.File, which I could then call in my compose file with:

- traefik.http.routers.test.middlewares=https-only@file.

If I create a chain in the compose file, I can then reference it in other routers/compose files, but doing it this way ties the middleware to a particular router/compose. For example, if I stop that router/compose the middleware would be removed. I feel like this is an intended use of the file property, but I'm having trouble getting it to work.

I'm calling the file in my traefik.toml as:

  [providers.file]
    ##Provide a dedicated file, or a directory of several dedicated files
##    filename = "/etc/traefik/rules/middlewares.toml"
    directory = "/etc/traefik/rules"
    watch = true

At first I was getting the error:

"Cannot start the provider *file.Provider: toml: cannot load TOML value of type string into a Go boolean"

So I edited my middlewares.toml trying to follow the docs:

[http.middlewares]
  {{ range $i, $e := until 100 }}
  [http.middlewares.https-only.redirectScheme{{ $e }}]
    scheme = "https-{{ $e }}"
  {{ end }}

but I'm now getting the error:

error msg="middleware \"https-only@file\" does not exist" entryPointName=https routerName=traefik@docker

I feel like modifying the middlewares.toml with Go templating per the docs helped the file to be read, but I'm still not having any luck creating middlewares with the file property. I tried adding a fake router and service in my middlewares.toml, hoping that the middleware would be created, but wouldn't route any traffic, but didn't have luck there either.

Is anyone doing something similar?

I feel like the issue may have to do with my middlewares.toml file and not having the right context or Go templating.

redirectScheme is the type of the middleware, https-only is the name, so {{ $e }} must be in the name not in the type.

Something like that:

[http.middlewares]
  {{ range $i, $e := until 100 }}
  [http.middlewares.https-only{{ $e }}.redirectScheme]
    scheme = "https"
  {{ end }}

but you don't need to create multiple redirectScheme.
Only one is enough, you just need to reference it where you need to use it.

https://docs.traefik.io/v2.0/middlewares/overview/#provider-namespace

I believe that's what I'm trying to do, create a single redirectScheme middleware using the file provider and then reference it where I need to use it as https-only@file. I'm referencing it in this way:

- traefik.http.routers.traefik.middlewares=https-only@file

I've edited my middlewares.toml (file) as you mentioned but am still getting the same error.

You don't need to use Go template: create a simple middleware and use it:

[http.middlewares]
  [http.middlewares.https-only.redirectScheme]
    scheme = "https"
version: "3.3"

services:
  traefik:
    image: "traefik:v2.0.0-rc2"
    command:
      - "--log.level=DEBUG"
      - "--api=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.file=true"
      - "--providers.file.filename=/etc/traefik/rules/middlewares.toml"
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.rule=Host(`dash.localhost`)"
      - "traefik.http.routers.traefik.middlewares=https-only@file"

I'm able to create middleware in my docker-compose file, but I'm having trouble creating middleware using the file provider. I tried to describe my reasoning in the OP for setting things up this way - basically I don't want middleware that I will reference to be tied to a single container. I guess I'm assuming that if I want to remove that container, then the middleware would break for any of the other containers that are referencing it, right? I want the middleware to be separated by the file provider so that it is always able to be referred to regardless of which containers are running or stopped.

Do you know if this is how using middleware in the file provider is supposed to work, or would I need to include routers and a service in the file as well?

Also, are you able to use that compose script, along with file referenced in that way? I've only tried this on rc2 so far.

In my previous example, I forgot to mount middlewares.toml

[http.middlewares]
  [http.middlewares.https-only.redirectScheme]
    scheme = "https"
version: "3.3"

services:
  traefik:
    image: "traefik:v2.0.0-rc2"
    command:
      - "--log.level=DEBUG"
      - "--api=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.file=true"
      - "--providers.file.filename=/etc/traefik/middlewares.toml"
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./middlewares.toml:/etc/traefik/middlewares.toml"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.rule=Host(`dash.localhost`)"
      - "traefik.http.routers.traefik.middlewares=https-only@file"
Logs
$ docker-compose up
Recreating tem_traefik_1 ... done
Attaching to tem_traefik_1
traefik_1  | time="2019-09-08T23:19:54Z" level=info msg="Configuration loaded from flags."
traefik_1  | time="2019-09-08T23:19:54Z" level=info msg="Traefik version 2.0.0-rc2 built on 2019-09-03T20:10:11Z"
traefik_1  | time="2019-09-08T23:19:54Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"http\":{\"address\":\":80\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{}},\"traefik\":{\"address\":\":8080\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{}}},\"providers\":{\"providersThrottleDuration\":2000000000,\"docker\":{\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"swarmModeRefreshSeconds\":15000000000},\"file\":{\"watch\":true,\"filename\":\"/etc/traefik/middlewares.toml\"}},\"api\":{\"dashboard\":true},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"}}"
traefik_1  | time="2019-09-08T23:19:54Z" level=error msg="\nYou haven't specified the sendAnonymousUsage option, it will be enabled by default.\n"
traefik_1  | time="2019-09-08T23:19:54Z" level=info msg="\nStats collection is enabled.\nMany thanks for contributing to Traefik's improvement by allowing us to receive anonymous information from your configuration.\nHelp us improve Traefik by leaving this feature on :)\nMore details on: https://docs.traefik.io/v2.0/contributing/data-collection/\n"
traefik_1  | time="2019-09-08T23:19:54Z" level=debug msg="No default certificate, generating one"
traefik_1  | time="2019-09-08T23:19:55Z" level=info msg="Starting provider aggregator.ProviderAggregator {}"
traefik_1  | time="2019-09-08T23:19:55Z" level=debug msg="Start TCP Server" entryPointName=http
traefik_1  | time="2019-09-08T23:19:55Z" level=info msg="Starting provider *file.Provider {\"watch\":true,\"filename\":\"/etc/traefik/middlewares.toml\"}"
traefik_1  | time="2019-09-08T23:19:55Z" level=debug msg="Start TCP Server" entryPointName=traefik
traefik_1  | time="2019-09-08T23:19:55Z" level=info msg="Starting provider *docker.Provider {\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"swarmModeRefreshSeconds\":15000000000}"
traefik_1  | time="2019-09-08T23:19:55Z" level=debug msg="Configuration received from provider file: {\"http\":{\"middlewares\":{\"https-only\":{\"redirectScheme\":{\"scheme\":\"https\"}}}},\"tcp\":{},\"tls\":{}}" providerName=file
traefik_1  | time="2019-09-08T23:19:55Z" level=debug msg="No default certificate, generating one"
traefik_1  | time="2019-09-08T23:19:55Z" level=debug msg="Provider connection established with docker 19.03.1-ce (API 1.40)" providerName=docker
traefik_1  | time="2019-09-08T23:19:55Z" level=debug msg="Configuration received from provider docker: {\"http\":{\"routers\":{\"traefik\":{\"middlewares\":[\"https-only@file\"],\"service\":\"traefik_tem\",\"rule\":\"Host(`dash.localhost`)\"}},\"services\":{\"traefik_tem\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.21.0.2:80\"}],\"passHostHeader\":true}}}},\"tcp\":{}}" providerName=docker
traefik_1  | time="2019-09-08T23:19:55Z" level=debug msg="No entryPoint defined for this router, using the default one(s) instead: [http traefik]" routerName=traefik@docker
traefik_1  | time="2019-09-08T23:19:55Z" level=debug msg="Creating middleware" routerName=traefik@docker serviceName=traefik_tem entryPointName=http middlewareName=pipelining middlewareType=Pipelining
traefik_1  | time="2019-09-08T23:19:55Z" level=debug msg="Creating load-balancer" entryPointName=http routerName=traefik@docker serviceName=traefik_tem
traefik_1  | time="2019-09-08T23:19:55Z" level=debug msg="Creating server 0 http://172.21.0.2:80" entryPointName=http routerName=traefik@docker serviceName=traefik_tem serverName=0
traefik_1  | time="2019-09-08T23:19:55Z" level=debug msg="Added outgoing tracing middleware traefik_tem" entryPointName=http routerName=traefik@docker middlewareName=tracing middlewareType=TracingForwarder
traefik_1  | time="2019-09-08T23:19:55Z" level=debug msg="Creating middleware" entryPointName=http routerName=traefik@docker middlewareName=https-only@file middlewareType=RedirectScheme
traefik_1  | time="2019-09-08T23:19:55Z" level=debug msg="Setting up redirection to https " middlewareType=RedirectScheme entryPointName=http routerName=traefik@docker middlewareName=https-only@file
traefik_1  | time="2019-09-08T23:19:55Z" level=debug msg="Adding tracing to middleware" entryPointName=http routerName=traefik@docker middlewareName=https-only@file
traefik_1  | time="2019-09-08T23:19:55Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=http middlewareName=traefik-internal-recovery
traefik_1  | time="2019-09-08T23:19:55Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=traefik middlewareName=traefik-internal-recovery
traefik_1  | time="2019-09-08T23:19:55Z" level=debug msg="No default certificate, generating one"
API
{
  "routers": {
    "traefik@docker": {
      "middlewares": [
        "https-only@file"
      ],
      "service": "traefik_tem",
      "rule": "Host(`dash.localhost`)",
      "status": "enabled",
      "using": [
        "http",
        "traefik"
      ]
    }
  },
  "middlewares": {
    "https-only@file": {
      "redirectScheme": {
        "scheme": "https"
      },
      "status": "enabled",
      "usedBy": [
        "traefik@docker"
      ]
    }
  },
  "services": {
    "traefik_tem@docker": {
      "loadBalancer": {
        "servers": [
          {
            "url": "http://172.21.0.2:80"
          }
        ],
        "passHostHeader": true
      },
      "status": "enabled",
      "usedBy": [
        "traefik@docker"
      ],
      "serverStatus": {
        "http://172.21.0.2:80": "UP"
      }
    }
  }
}

The problem seems to have been that I had an additional middleware defined in the middlewares.toml that I'm assuming was incorrectly formatted. When I removed everything and set it up (without Go templating!) everything seems to work and I can see it pick up the config in the logs as it shows in yours. I'm not sure what command you're using to view the API information, but I'm assuming mine would be the same since I'm no longer getting errors.

It seems I need to figure out the correct formatting for the headers in a toml file :upside_down_face:

Thanks a lot for your help.