Global HTTPS redirection issues with other Docker Stack files

mindgonemad · June 11, 2023, 1:06am

I was having issues with global redirection but not always and finally figured out it's related to having services in different Docker Stack files (single Docker Swarm node). Any services added to the same Stack file as Traefik all redirect from http to https without issue, whilst services that are spun up from a seperate Stack file do not redirect, so https works fine while http times-out (page not responding).

I have tried setting up redirect using entrypoints, middlewares and older style of labels and all give the same result.

I have spent days on this, run I don't know how many searches and have finally given up. I'm hoping someone has an easy fix I have just missed or do I have to remove the global redirect and type a heap of repetitive code to redirect for every service?

Looking forward toi any help that can be provided.

Thanks

bluepuma77 · June 11, 2023, 8:33am

You want a global http to https redirect and it‘s not working for you? Have you placed it on the entrypoint like the simple Traefik example? Works for me.

Share your Traefik static and dynamic config, and docker-compose.yml if used.

mindgonemad · June 11, 2023, 10:43pm

docker-stack-traefik-v3.yml

version: "3.8"

volumes:
  letsencrypt:
  logs:

networks:
  traefik-public:
    external: true

services:
  traefik:
    image: traefik:3.0.0-beta2
    networks:
      - traefik-public
    command:
      # Enable Docker swarm; Enable Docker in Traefik
      - --providers.docker.swarmMode=true
      - --providers.docker.exposedbydefault=false
      - --providers.docker.network=traefik-public
      # Entrypoints and Global Redirection (http->https) 
      - --entrypoints.web.address=:80
      - --entrypoints.web.http.redirections.entrypoint.to=websecure
      - --entrypoints.web.http.redirections.entrypoint.scheme=https
      - --entrypoints.web.http.redirections.entrypoint.permanent=true
      - --entrypoints.websecure.address=:443
      # Global TLS (Let's Encrypt)
      - --entrypoints.websecure.http.tls=true
      # Let's Encrypt SSL
      - --certificatesresolvers.leresolver.acme.email=<EMAIL>
      - --certificatesresolvers.leresolver.acme.storage=/letsencrypt/acme.json
      - --certificatesresolvers.leresolver.acme.tlschallenge=true
      # Enable dashboard
      - --api.dashboard=true
      # Logging (By default, the level is set to ERROR. Alternative levels are DEBUG, PANIC, FATAL, ERROR, WARN, and INFO)
      - --log.level=ERROR
      - --log.format=json
      - --log.filePath=/logs/traefik.log
      - --accesslog.filePath=/logs/traefik-access.log
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - letsencrypt:/letsencrypt
      - logs:/logs
      # Let Traefik listen to the Docker events
      - /var/run/docker.sock:/var/run/docker.sock:ro
    deploy:
      placement:
        constraints:
          - node.role == manager
      labels:
        traefik.enable: "true" # Required if containers not exposed by default
        # Global redirection: https://www -> https://
        # traefik.http.routers.wwwsecure-catchall.rule: hostregexp(`{host:(www\.).+}`)
        # traefik.http.routers.wwwsecure-catchall.entrypoints: websecure
        # traefik.http.routers.wwwsecure-catchall.tls: "true"
        # traefik.http.routers.wwwsecure-catchall.middlewares: wwwtohttps
        ## Redirection middleware: https://www -> https:// (http->https redirection at entrypoint level)
        # traefik.http.middlewares.wwwtohttps.redirectregex.regex: ^https?://(?:www\.)?(.+)
        # traefik.http.middlewares.wwwtohttps.redirectregex.replacement: https://$${1}
        # traefik.http.middlewares.wwwtohttps.redirectregex.permanent: "true"
        # UI Dashboard
        traefik.http.routers.traefik.rule: "Host(`<DOMAIN>`)"
        traefik.http.routers.traefik.service: api@internal
        traefik.http.routers.traefik.middlewares: auth-traefik
        traefik.http.services.traefik.loadbalancer.server.port: 8080 # Port used by service (Docker Swarm requirement, defined in image)
        ## Basic Auth
        traefik.http.middlewares.auth-traefik.basicauth.users: "<USER>:<AUTHCODE>"

  whoami:
    image: containous/whoami:v1.3.0
    networks:
      - traefik-public
    deploy:
      labels:
        traefik.enable: "true" # Required if containers not exposed by default
        # Host & Entrypoint
        traefik.http.routers.whoami.rule: "Host(`traefik-whoami.<DOMAIN>`)"
        traefik.http.routers.whoami.middlewares: auth-whoami
        traefik.http.services.whoami.loadbalancer.server.port: 80 # Port used by service (Docker Swarm requirement, defined in image)
        # Let's Encrypt (TLS)
        traefik.http.routers.whoami.tls: "true"
        traefik.http.routers.whoami.tls.certresolver: leresolver
        ## Basic Auth Middleware
        traefik.http.middlewares.auth-whoami.basicauth.users: "<USER>:<AUTHCODE>"

docker-stack-whoami.yml

version: "3.9"

volumes: # Docker Volumes - Using instead of bind mounts, data can be accessed by FTP'ing in via secondary container.
  grav-user:
  grav-config:

networks:
  traefik-public:
    external: true

services:
  whoami:
    image: containous/whoami:v1.3.0
    networks:
      - traefik-public
    deploy:
      labels:
        traefik.enable: "true" # Required if containers not exposed by default
        # Host & Entrypoint
        # traefik.http.routers.whoami-abk-grav.entrypoints: websecure
        traefik.http.routers.whoami-abk-grav.rule: "Host(`whoami.<DOMAIN>`)"
        traefik.http.routers.whoami-abk-grav.middlewares: auth-whoami-abk-grav
        traefik.http.services.whoami-abk-grav.loadbalancer.server.port: 80 # Port used by service (Docker Swarm requirement, defined in image)
        # Let's Encrypt (TLS)
        traefik.http.routers.whoami-abk-grav.tls: "true"
        traefik.http.routers.whoami-abk-grav.tls.certresolver: leresolver
        ## Basic Auth Middleware
        traefik.http.middlewares.auth-whoami-abk-grav.basicauth.users: "<USER>:<AUTHCODE>"

"Who Am I" on Traefik redirects correctly from http to https, whiklst any services I create using a seperate Docker Stack don't. They all work on https but time-out when I try http. I have only listed the Who Am I service as an example but there are more.

I wonder if Docker appending stack names to things has something to do with it, eg. the logs volume actually gets named traefik_logs, yet the https routing works fine... no idea.

Anyway, i've been scratching my head for so long I am completely out of ideas as have tried using different examples of global redirects using labels and they all fail as well.

bluepuma77 · June 12, 2023, 5:29am

Yes, docker compose renames things, you need to use name: or external: true for the Docker network to avoid that.

You enable TLS on the router labels, but without setting a dedicated entrypoint. So Traefik enables it for all: web and websecure.

mindgonemad · June 12, 2023, 8:56am

Thanks for responding, but due to my ineptness I am still stumped.

Yes, docker compose renames things, you need to use name: or external: true for the Docker network to avoid that.

Do I need to worry about this, as in will it affect the result i'm after or just good to know for management as I don't want to have to go renaming volumes at this stage unless I have to?

You enable TLS on the router labels, but without setting a dedicated entrypoint. So Traefik enables it for all: web and websecure .

I added

traefik.http.routers.whoami-abk-grav.entrypoints: websecure

but then I get a time-out on http as before but https now gives me a 404, so it's even more broken. I tried replacing websecure with web and also tried web, websecure and they didn't work either.

I'm hoping you can offer a little more insight to a hack like me.

bluepuma77 · June 12, 2023, 10:39am

The first point was FYI only, as you have external set.

For me multiple stacks work fine with Docker Swarm and Traefik, for example with Portainer:

      labels:
        - "traefik.enable=true"
        - "traefik.docker.network=proxy"
        - "traefik.http.routers.portainer.entrypoints=websecure"
        - "traefik.http.routers.portainer.rule=Host(`portainer.example.com`)"
        - "traefik.http.services.portainer.loadbalancer.server.port=9000"
        - "traefik.http.services.portainer.loadbalancer.passhostheader=true"

Note that traefik-whoami also enables TLS on http, which might lead to unexpected results.

mindgonemad · June 13, 2023, 6:31am

It doesn't make any sense to me, if I match it to your code its still broken:
pasting in a bare url (without https, www, etc) gets directed to https and works.
Changing it to http and I get the time out.
It's the same result for both whoami and the Grav website.

Can I ask what the go with the passhostheader is, is that a portainer thing as I added it and had no difference either way that I noticed?

Thanks for your help so far. Maybe it's just like the rest of life, where shit just malfunctions around me .

bluepuma77 · June 13, 2023, 6:52am

Did you fix your traefik-whoami to only use entrypoint websecure?

By default, passHostHeader is true.

So this can be ignored

mindgonemad · June 13, 2023, 8:02am

Did you fix your traefik-whoami to only use entrypoint websecure ?

For a short period I was building my stack from the wrong file (now fixed), so adding websecure (as below) works for https but http still times-out (along with the Grav service).

traefik.http.routers.grav-abk.entrypoints: websecure

bluepuma77 · June 13, 2023, 9:19am

This does not make sense. You open the port on compose and you have an entrypoint for it. So it should work. Do you have a Loadbalancer or Firewall in front of Traefik?

You set redirections.entrypoint.permanent=true, so your normal browser will not use http again, instead it will remember to use https.

I still suspect you have SSL enabled on port 80, maybe test it:

openssl s_client -showcerts -connect example.com:80

mindgonemad · June 13, 2023, 10:19am

Do you have a Loadbalancer or Firewall in front of Traefik?

No load balancer and as for firewall, I was about to say probably noy, but guessing I would be wrong... I run Alpine Linux in a Linode years ago.

I just looked in Linode and noticed I have some firewalls through them:
Port 22 All IPv4, All IPv6
Port 53 All IPv4, All IPv6
Port 976 All IPv4, All IPv6
Port 443 All IPv4, All IPv6
When I list iptables I get:
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-INGRESS
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-N f2b-sshd
-A INPUT -p tcp -m multiport --dports 22 -j f2b-sshd
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-INGRESS
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o docker_gwbridge -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker_gwbridge -j DOCKER
-A FORWARD -i docker_gwbridge ! -o docker_gwbridge -j ACCEPT
-A FORWARD -i docker_gwbridge -o docker_gwbridge -j DROP
-A DOCKER-INGRESS -p tcp -m tcp --dport 443 -j ACCEPT
-A DOCKER-INGRESS -p tcp -m state --state RELATED,ESTABLISHED -m tcp --sport 443 -j ACCEPT
-A DOCKER-INGRESS -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER-INGRESS -p tcp -m state --state RELATED,ESTABLISHED -m tcp --sport 80 -j ACCEPT
-A DOCKER-INGRESS -j RETURN
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i docker_gwbridge ! -o docker_gwbridge -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o docker_gwbridge -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
-A f2b-sshd -j RETURN

Response from running "openssl s_client -showcerts -connect example.com:80" in terminal:

CONNECTED(00000003)

139766762289992:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:ssl/record/ssl3_record.c:331:

no peer certificate available

No client certificate CA names sent

SSL handshake has read 5 bytes and written 313 bytes

Verification: OK

New, (NONE), Cipher is (NONE)

Secure Renegotiation IS NOT supported

No ALPN negotiated

Early data was not sent

Verify return code: 0 (ok)

I hope some of this makes sense to you as it is all gobbledygook to me .

bluepuma77 · June 13, 2023, 11:01am

FW could be ok, seems no SSL on port 80.

So what does this do: curl -v http://yourdomain.com:80

mindgonemad · June 13, 2023, 9:03pm

curl -v http://grav.mydomain.com.au:80

*Trying 172.105.179.94:80...
*Connected to grav.mydomain.com.au (172.105.179.94) port 80 (#0)
GET / HTTP/1.1
Host: grav.mydomain.com.au
User-Agent: curl/8.0.1
Accept: /

HTTP/1.1 301 Moved Permanently
Location: https://grav.mydomain.com.au/
Date: Tue, 13 Jun 2023 20:59:26 GMT
Content-Length: 17

*Connection #0 to host grav.mydomain.com.au left intact Moved Permanently

bluepuma77 · June 14, 2023, 4:47am

So it works, you get a redirect to https.

mindgonemad · June 14, 2023, 10:12am

If I enter grav.infinityabk.com.au into a modern browser then it successfully gives me https://grav.infinityabk.com.au/, but if I take the s of https so it is http://grav.infinityabk.com.au/ then I get a timeout rather than redirecting... or am I misunderstanding something fundamental?

bluepuma77 · June 14, 2023, 6:53pm

Curl is a command line browser, you could connect to http and get the status 301 redirect to https. Not sure what the problem with your browser is.

You can try curl --location http://domain to make it follow redirects.

mindgonemad · June 16, 2023, 4:01am

Sorry about the delay in getting back to you. Something weird is going on and I have no idea what. Last night I ran that curl --location http://grav.infinityabk.com.au command and it timed out, so I went back and ran curl -v http://grav.infinityabk.com.au:80 again and this time it timed out (no server changes since the other day). I never got around to writing a response till now so tried them again and now they are working again... WTF?

If I paste grav.infinityabk.com.au in multiple browsers the response is:

Arc Browser: Takes me to https (works).
Firefox (Mac): Timed out. If i manually add https in front it works.
Safari (Mac & iOS): Timed out. If i manually add https in front it works.

I'm guessing Arc automatically switches it to https while Firefox and Safari don't.

Why would Curl work but the browsers not? Are you as stumped as me?

bluepuma77 · June 16, 2023, 6:43am

To me the results seem inconsistent.

Do you have multiple DNS entries for the domain? Browsers will pick randomly one IP.
Do you have a load balancer in front of Traefik? What are the targets?
Do you run a Docker overlay network over a VLAN and the MTU does not match, breaking larger TCP packets?

mindgonemad · June 17, 2023, 4:14am

1. Do you have multiple DNS entries for the domain? Browsers will pick randomly one IP.

I don't believe so. I manage the domain on Linode and there are two a records, one is "abkbookkeeping.com.au" and the other is "*", both pointing to the same server IP address.

2. Do you have a load balancer in front of Traefik? What are the targets?

I haven't set up any load balancers on Linode. As for Docker i'm running a single node swarm and have no idea if that has any sort of load balancing but doubt it.

3. Do you run a Docker overlay network over a VLAN and the MTU does not match, breaking larger TCP packets?

Docker uses an overlay network for Traefik but no idea about the VLAN or MTU that you mention... guessing not.

Topic		Replies	Views
Global Http to Https Redirect not working Traefik v2 docker-swarm , middleware	4	6602	March 6, 2022
Global HTTP->HTTPS redirect on Docker, TLS config not picked up Traefik v2 docker	2	1310	January 22, 2020
Multiple levels of redirects do not work Traefik v2 docker-swarm	5	101	April 29, 2024
Global https redirect Traefik v2 docker	5	670	December 7, 2020
Problem with http -> https redirection Traefik v2 docker-swarm	1	2193	January 12, 2020

Global HTTPS redirection issues with other Docker Stack files

Related topics