Getting "Bad Gateway"/404 page when supposed to route to container port 8443

Traefik Traefik v2
1

I'm trying to get Crafty Controller working so I can host some Minecraft servers for my friends. I've been able to verify that all my containers are running properly, but when I try accessing the dashboard at crafty.mydomain.com I get a page that says "Bad Gateway." If I access the dashboard from crafty.mydomain.com:8443 (the default port for the secure dashboard) I got the dashboard with its default self-signed certificate.

Or I did when the problem first manifested. While double-checking things before posting this I'm now getting a 404 page at the bare subdomain and all browsers I try are rejecting the self-signed certificate (I could click through to the portal in some browsers before).

Clearly I'm missing something in my routing/networking config, but I'm enough of a newbie to not have it be obvious to me.

My traefik.yml:

api:
  dashboard: true

serversTransport:
  insecureSkipVerify: true

entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websecure

  websecure:
    address: ":443"
    http:
      middlewares:
        - secureHeaders@file
      tls:
        certResolver: letsencrypt

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
  file:
    filename: /configurations/dynamic.yml

certificatesResolvers:
  letsencrypt:
    acme:
      email: myemail@provider.com
      storage: acme.json
      keyType: EC384
      httpChallenge:
        entryPoint: web

My dynamic.yml:

# Dynamic configuration
http:
  middlewares:
    secureHeaders:
      headers:
        sslRedirect: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 31536000
    user-auth:
      basicAuth:
        users:
          - "redacted:redacted"

tls:
  options:
    default:
      cipherSuites:
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
      minVersion: VersionTLS12

My Traefic docker-compose.yml

version: "3.8"

services:
  traefik:
    image: "traefik:latest"
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - "no-new-privileges:true"
    networks:
      - proxy
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "/etc/localtime:/etc/localtime:ro"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./data/traefik.yml:/traefik.yml:ro"
      - "./data/acme.json:/acme.json"
      - "./data/configurations:/configurations"
    labels:
      - traefik.enable=true
      - traefik.docker.network=proxy
      - traefik.http.routers.traefik-secure.entrypoints=websecure
      - traefik.http.routers.traefik-secure.rule=Host(`traefik.mydomain.com`)
      - traefik.http.routers.traefik-secure.service=api@internal
      - traefik.http.routers.traefik-secure.middlewares=user-auth@file
networks:
  proxy:
    external: true

And my Crafty Controller docker-compose.yml:

version: '3'

services:
  crafty:
    container_name: crafty_container
    image: registry.gitlab.com/crafty-controller/crafty-4:4.2.1
    restart: unless-stopped
    environment:
        - TZ=redacted
    ports:
        - "8000:8000" # HTTP
        - "8443:8443" # HTTPS
        - "8123:8123" # DYNMAP
        - "19132:19132/udp" # BEDROCK
        - "25500-25600:25500-25600" # MC SERV PORT RANGE
    volumes:
        - ./backups:/crafty/backups
        - ./logs:/crafty/logs
        - ./servers:/crafty/servers
        - ./config:/crafty/app/config
        - ./import:/crafty/import
    labels:
      - traefik.enable=true
      - traefik.docker.network=proxy
      - "traefik.http.routers.crafty.tls=true"
      - "traefik.http.routers.crafty.tls.certresolver=letsencrypt"
      - "traefik.http.routers.crafty.service=crafty"
      - traefik.http.routers.crafty.entrypoints=websecure
      - "traefik.http.services.crafty.loadbalancer.server.scheme=https"
      - "traefik.http.services.crafty.loadbalancer.server.port=8443"
      - traefik.http.routers.crafty.rule=Host(`crafty.mydomain.com`)
      - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto = https"
    networks:
      - proxy

networks:
  proxy:
    external: true

I'm not using swarm mode, and a lot of the fancier settings are pulled from a tutorial I found, with a bit of tweaking.

This feels like it should be an easy fix, but I'm neither skilled enough at parsing the docs nor experienced enough with Traefik in general to know what I should do here. I'd ask the Crafty Controller folks, but the devs claim that none of them use Traefik themselves and both the install docs and reverse proxy configuration docs look like they should work (the Traefik config was provided by a member of their community), but clearly aren't working with how I attempted to merge them with my existing Traefik config.

So, what am I doing wrong? Hopefully it's something simple and/or obvious, but I seem to have reached the limits of my knowledge. All help is appreciated, thanks. And sorry for the wall of text, but I'm not sure what information is unnecessary for this problem, hence me providing everything I currently can.

2

Usually Traefik is configured to terminate TLS, then internally use unencrypted requests, mostly going to port 80.

You are trying to force Traefik to use https/TLS internally, so you need to make sure that Traefik accepts the (probably custom created) cert of the target service/container. This can be worked around with insecureSkipVerify.

Enable and check Traefik debug log and dashboard.

3

Appreciate the explanation. I think I already have that enabled in my static configuration (although I'll probably try to get it set up only for this service after getting this working). Did I somehow fail at that? Do I need to change how I'm trying to accomplish this?

The current top-level problem (and why I'm now getting 404 instead of Bad Gateway) appears to be that it doesn't like my host routing rule:

error while parsing rule Host(crafty.mydomain.com): crafty.mydomain.com is not defined

and

invalid rule Host(crafty.mydomain.com), error: crafty.mydomain.com is not defined

are the errors in the routers section of my dashboard. I'm also confused by this, since that part looks like it should be fine. I did wrap the rest of the labels section in double quotes just in case, but nothing seems to have changed after I did that. Am I missing something obvious again, or have I somehow managed to get Traefik to bungle something it should normally be fine with?

Thanks again. At least I'm getting to learn things with this project that go beyond what I normally encounter.

4

Strange, seems to be correct. Within Host() you need to use backticks, this is visible here if you use 3 backticks before and after code.

5

I'm seeing it correctly in the blocks of my original post (i.e., all host directives have the backticks and display as such in the code blocks). Also, Firefox once again allows me to get to the dashboard through port 8443 and port 8000 gets auto-routed to 8443, so things aren't completely broken.

I'm starting to think it might be easier to re-think how everything is trying to work. I'm dedicated to getting things working; I'm not dedicated to making the setup work as suggested by the project's documentation. I'm stealing a few moments from work to post this update, so I won't be able to try any changes for a few hours, but when I get around to trying something different, the first thing to try would be routing to port 8000 instead of 8443, yes? That should theoretically make insecureSkipVerify unnecessary, and maybe even make it less likely that Traefik breaks things it shouldn't be breaking?

6

All you should need is

    labels:
      - traefik.enable=true
      - traefik.docker.network=proxy
      - traefik.http.routers.crafty.entrypoints=websecure
      - traefik.http.routers.crafty.tls.certresolver=letsencrypt
      - traefik.http.routers.crafty.rule=Host(`crafty.mydomain.com`)
      - traefik.http.routers.crafty.service=crafty
      - traefik.http.services.crafty.loadbalancer.server.port=8080
7

Well, making those changes made the Traefik routing error go away, but the page hangs and fails to load, eventually timing out.

Apparently rethinking how I do things is explicitly against Crafty's design. Per their docs on accessing the panel:

Crafty makes use of HTTPS to protect your personal data. When accessing Crafty you must explicitly type https when entering Crafty’s address in your browser.
Crafty can be accessed via localhost , 127.0.0.1 , and your machine’s IP address ,
but again, you must type https:// before the address.
You may recieve a warning screen in your browser notifying "Your connection is not private", don't worry it is, it's just because crafty by default uses a self-signed cert, which your browser can't verify against the big certificate authorities.
Just click the dropdown and proceed anyway!

I'm not sure where to go from here, except perhaps to look for another, more Traefik-friendly solution.

I'm out of what I'd call reasonable ideas (I'm pretty sure trying to get Traefik to play nice with another reverse proxy that plays better with Crafty Controller is not reasonable), and I'm not sure how to convince the devs that having a more generally reverse-proxy-friendly design (at least an option to have traffic unencrypted when behind a reverse proxy) is a good idea. If you have any other ideas I'm all ears.

Thanks very much for your help, though. I've learned some things, and I'll probably be able to use them with the next solution I try.

8

There is an unwritten standard for web applications that want https: they either see a request via https or accept something like a X-Forward-Scheme=https header in http for exactly the reverse proxy use case (which Traefik automatically sets).

If your app doesn’t follow this, then you might want to forward requests with https. As the app will probably just create a random cert, you need to tell Traefik to trust the cert, either by configuring the cert or telling Traefik to trust any cert when forwarding.

This can be done with insecureSkipVerify. It can be easily defined globally in your Traefik static config or you can create a serverTransport in a dynamic config file (not Docker labels) and assign this transport to a dedicated service only.

And set according scheme/port (Docker label) or URL (file) in dynamic config for the target.

9

Perhaps there's been progress???

I've got insecureSkipVerify configured in my dynamic.yml:

http:
  services:
    crafty:
      loadBalancer:
        serversTransport: craftyTransport

  serversTransports:
    craftyTransport:
      insecureSkipVerify: true

I think that should be correct, if I've parsed the info correctly.

I also got debug logs working and am now getting

"'500 Internal Server Error' caused by: tls: failed to verify certificate: x509: cannot validate certificate for <IP> because it doesn't contain any IP SANs"

So perhaps I don't have insecureSkipVerify configured correctly after all?

At least it's not the same errors anymore.

10

Is that an error from client to Traefik or from Traefik to target service?

What does your request to Traefik look like?

11

I'm not sure I understand the question. What do you mean by my request to Traefik? The headers, etc. from the browser or something else? If something else, how do I find it to check?

Thanks.

12

I just mean the URL of your request.

13

That would be https://crafty.mydomain.com although I am usually only trying crafty.mydomain.com directly in the address bar.