Trying to migrate from v1 to v2, some services are working while others do not with similar configurations

Traefik Traefik v2
1

I am migrating my Traefik config from v1 to v2 today, and ran into this problem. Someof my rules are working while others are not, yet the configuration seems to be the same.

My global traefik config:

global:
  sendAnonymousUsage: true
entryPoints:
  http:
    address: :80
  https:
    address: :443
providers:
  providersThrottleDuration: 2s
  docker:
    watch: true
    endpoint: unix:///var/run/docker.sock
    swarmModeRefreshSeconds: 15s
api:
  insecure: true
  dashboard: true
log:
  level: INFO
accessLog:
  filters:
    statusCodes: [400-599]
    retryAttempts: true
certificatesResolvers:
  letsencrypt:
    acme:
      email: [REDACTED]
      storage: acme.json
      httpChallenge:
        entryPoint: http

Working configurations

these configurations work

Airsonic

# docker-compose.yml
# ...
labels:
  traefik.enable: "true"
  traefik.http.routers.airsonic.tls: 'true'
  traefik.http.routers.airsonic.tls.certresolver: letsencrypt
  traefik.http.routers.airsonic.rule: Host(`music.tams.tech`,`music.madscientists.co`)
  traefik.http.services.airsonic.loadbalancer.server.port: 4040
  traefik.http.services.airsonic.loadbalancer.passhostheader: "true"
  traefik.http.middlewares.airsonic-csp.headers.customresponseheaders.Content-Security-Policy: "
    default-src 'self';
    script-src 'self' 'unsafe-inline' 'unsafe-eval' www.gstatic.com;
    img-src 'self' *.akamaized.net;
    style-src 'self' 'unsafe-inline' fonts.googleapis.com;
    font-src 'self' fonts.gstatic.com;
    frame-src 'self';
    object-src 'none';"
# ...

Jellyfin

# docker-compose.yaml
# ...
labels:
  traefik.enable: "true"
  traefik.http.routers.mded.tls: "true"
  traefik.http.routers.mded.tls.certresolver: letsencrypt
  traefik.http.routers.mded.rule: Host(`mded.tams.tech`,`mded.madscientists.co`)
  traefik.http.services.mded.loadbalancer.server.port: 3000
# ...

There are others that work as well

NOT Working

these configurations seem similar to me but mysteriously don't work

Nextcloud

labels:
  traefik.enable: "true"
  traefik.http.routers.nextcloud.tls: 'true'
  traefik.http.routers.nextcloud.tls.certresolver: letsencrypt
  traefik.http.routers.nextcloud.rule: Host(`cloud.tams.tech`,`cloud.madscientists.co`)
  traefik.http.services.nextcloud.loadbalancer.server.port: 80
  tech.tams.dns_hosts: cloud.madscientists.co cloud.tams.tech

Draw.io

labels:
  traefik.enable: "true"
  traefik.http.routers.draw_io.tls: 'true'
  traefik.http.routers.draw_io.tls.certresolver: letsencrypt
  traefik.http.routers.draw_io.rule: Host(`draw.tams.tech`,`draw.madscientists.co`)
  traefik.http.services.draw_io.loadbalancer.server.port: 8080

What happens

When making a request to any of the "not working" servers, a "Gateway Timeout" is eventually received. Full output of a verbose curl invocation:

*   Trying [REDACTED]:443...
* TCP_NODELAY set
* Connected to cloud.tams.tech ([REDACTED]) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=cloud.madscientists.co
*  start date: Dec 14 16:51:43 2019 GMT
*  expire date: Mar 13 16:51:43 2020 GMT
*  subjectAltName: host "cloud.tams.tech" matched cert's "cloud.tams.tech"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55dae7744740)
> GET / HTTP/2
> Host: cloud.tams.tech
> user-agent: curl/7.67.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 504 
< content-type: text/plain; charset=utf-8
< content-length: 15
< date: Sat, 14 Dec 2019 18:44:35 GMT
< 
* Connection #0 to host cloud.tams.tech left intact
Gateway Timeout

Additionally, requests made directly to the container itself work as expected -- for example, invoking curl -v -HHost:cloud.tams.tech http://<container IP>/login returns the actual login page.

Edit:

Between when I posted this and now, the draw.io configuration began to work.

2

To me, it looks like it could be network issue. Are Traefik and the non-working services on the same overlay network? Can they reach it other?

As the configuration just began to work, I suspect it might not be an immediate issue with Traefik.

3

All containers (that are intended to be exposed) are configured like this:

services:
  service:
    # ...
    networks:
      - web
# ...
networks:
  web:
    external: true

Running the following command returns a list of networks with different aliases but the same Gateway value. Having the same gateway address means they must be the same network, even if they aren't aliased to the same thing.

for container in `docker ps -q | tr '\n' ' '`; do
    docker inspect --format="{{.NetworkSettings.Networks.web.Aliases}}" $container
    docker inspect --format="{{.NetworkSettings.Networks.web.Gateway}}" $container
done

Is there something I'm missing here?

4

Yes. The following command succeeds

docker exec -it traefik wget -O - --header 'Host: cloud.tams.tech' "`docker inspect --format="{{.NetworkSettings.Networks.web.IPAddress}}" nextcloud_mad_cloud_1`/login"
7

Sorry, I was on vacation. Does the issue still persist?

8

It looks like the issues were with applications which used http instead of https for default protocol of generated links. I have worked around this by configuring all my applications to link to the https equivalents instead, and things seem to be working now as long as I type https explicitly. However, visiting any site as the regular http url returns a standard 404 Not found message rather than redirecting. For example http://draw.tams.tech vs https://draw.tams.tech.

9

Yeah, that is intended. As soon as you have TLS configured for a router, its just using that. You will need a second router to redirect from HTTP to HTTPS.