418 response codes; requests only getting handled by noop@internal router

adunne09 · November 16, 2023, 12:34am

Hello,

I am attempting to use traefik to reverse-proxy web services. I have followed the Ibracorp guide, and believe that I have done everything correctly. I am using Cloudflare as my registrar and DNS provider, and all of my containers are using the same custom docker network.

I am not seeing any problems in the logs. All I am trying to accomplish initially right now is to access https://traefik.zerosync.co, but the only response I have gotten so far is 418; upon doing some research, I believe that means that my requests are getting processed by the noop@internal router, but I have no iea where to go from there.

Any help would be much appreciated. Let me know if there's any other information I can provide that would help. Thanks

bluepuma77 · November 16, 2023, 5:37am

Share your full Traefik static and dynamic config, and docker-compose.yml if used.

adunne09 · November 16, 2023, 5:27pm

traefik.yml:

global:
  checkNewVersion: true
  sendAnonymousUsage: false
serversTransport:
  insecureSkipVerify: true

entryPoints:
  http:
    address: :80
    forwardedHeaders:
      trustedIPs: &trustedIps
        # Start of Clouflare public IP list for HTTP requests, remove this if you don't use it
        - 173.245.48.0/20
        - 103.21.244.0/22
        - 103.22.200.0/22
        - 103.31.4.0/22
        - 141.101.64.0/18
        - 108.162.192.0/18
        - 190.93.240.0/20
        - 188.114.96.0/20
        - 197.234.240.0/22
        - 198.41.128.0/17
        - 162.158.0.0/15
        - 104.16.0.0/13
        - 104.24.0.0/14
        - 172.64.0.0/13
        - 131.0.72.0/22
        - 2400:cb00::/32
        - 2606:4700::/32
        - 2803:f800::/32
        - 2405:b500::/32
        - 2405:8100::/32
        - 2a06:98c0::/29
        - 2c0f:f248::/32
        # End of Cloudlare public IP list
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https

  https:
    address: :443
    forwardedHeaders:
      # Reuse list of Cloudflare Trusted IP's above for HTTPS requests
      trustedIPs: *trustedIps
    http:
      tls:
        # Generate a wildcard domain certificate
        certResolver: letsencrypt
        domains:
          - main: zerosync.co
            sans:
              - '*.zerosync.co'
      middlewares:
        - securityHeaders@file

providers:
  providersThrottleDuration: 2s

  file:
    filename: /etc/traefik/fileConfig.yml
    watch: true

  docker:
    watch: true
    network: deadlift    # Add Your Docker Network Name Here
    # Default host rule to containername.domain.example
    defaultRule: "Host(`{{ lower (trimPrefix `/` .Name )}}.zerosync.co`)"    # Replace with your domain
    swarmModeRefreshSeconds: 15s
    exposedByDefault: false
    # endpoint: "tcp://dockersocket:2375"

api:
  dashboard: true
  insecure: true

log:
  level: DEBUG

certificatesResolvers:
  letsencrypt:
    acme:
      email: alex.dunne@zerosync.co # change to admin?
      storage: /etc/traefik/acme.json
      dnsChallenge:
        provider: cloudflare
        # Used to make sure the dns challenge is propagated to the rights dns servers
        resolvers:
          - "1.1.1.1:53"
          - "1.0.0.1:53"

accessLog: {}

fileConfig.yml:

http:
  middlewares:
    # Security headers
    securityHeaders:
      headers:
        customResponseHeaders:
          X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex"
          X-Forwarded-Proto: "https"
          server: ""
        customRequestHeaders:
          X-Forwarded-Proto: "https"
        sslProxyHeaders:
          X-Forwarded-Proto: "https"
        referrerPolicy: "same-origin"
        hostsProxyHeaders:
          - "X-Forwarded-Host"
        contentTypeNosniff: true
        browserXssFilter: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsSeconds: 63072000
        stsPreload: true

tls:
  options:
    default:
      minVersion: VersionTLS12
      cipherSuites:
        - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305

not using docker-compose

adunne09 · November 16, 2023, 5:29pm

this is the traefik log from when I receive a 418:

<IP> - - [16/Nov/2023:17:28:30 +0000] "GET / HTTP/1.1" 418 0 "-" "-" 301 "http-to-https@internal" "-" 0ms

bluepuma77 · November 16, 2023, 6:11pm

Format your config, use 3 backticks in front and after, in yaml ever space matters.

adunne09 · November 16, 2023, 6:51pm

done, thanks. Seeing any clues as to the source of this issue?

bluepuma77 · November 16, 2023, 7:26pm

What do you expect to see? Do you have any docker containers running for the provider.docker to be picked up?

I don’t think Traefik has a "Welcome to Traefik" page like Apache or nginx.

The 418 is a funny status code ("I’m a teapot"), the 301 would be the one matching a http to https redirect.

<ip> - - [16/Nov/2023:18:31:54 +0000] "GET / HTTP/1.1" 301 17 "-" "-" 526 "web-to-websecure@internal" "-" 0ms

You got a funny container with high routing priority running?

adunne09 · November 16, 2023, 7:50pm

I have one other service currently that I expect traefik to reverse proxy; it is a simple html nginx container called "home". I am running it with these labels:

traefik.enable=true
traefik.http.routers.home.entryPoints=https
traefik.http.routers.home.rule=Host(`zerosync.co`) || Host(`www.zerosync.co`)

Both traefik and home are on the same custom docker network.

I have a port mapping to enable the home container on my local network, and navigating to that IP and port does return what I expect to see when I navigate to the actual domains specified in the rule.

Does that help at all?

bluepuma77 · November 16, 2023, 8:08pm

I highly recommend to remove the single quotes around key and value.

You are running Traefik in Docker container, but start it manually from CLI?

adunne09 · November 16, 2023, 8:16pm

Changed the format of the label kv pairs.

no, I am starting everything through the Unraid UX that facilitates adding Docker containers.

bluepuma77 · November 16, 2023, 8:24pm

Recommend to also remove spaces.

So you have not other containers? I doubt that a 418 status code is from Traefik. It shouldn’t come from Cloudflare, as you see entries in Traefik access. So it can only be from another target.

Enable Traefik debug log and Traefik access log explicitly in JSON format, post both here again (start Traefik and do a request, use 3 backticks )

adunne09 · November 16, 2023, 8:41pm

I updated traefik.yml to output logs with the following config-

log:
  level: DEBUG
accessLog:
  format: json

This is the log when I made a request:

{"ClientAddr":"<ip>:28398","ClientHost":"<ip>","ClientPort":"28398","ClientUsername":"-","DownstreamContentSize":0,"DownstreamStatus":418,"Duration":60912,"OriginContentSize":0,"OriginDuration":0,"OriginStatus":0,"Overhead":60912,"RequestAddr":"www.zerosync.co","RequestContentSize":0,"RequestCount":13,"RequestHost":"www.zerosync.co","RequestMethod":"GET","RequestPath":"/","RequestPort":"-","RequestProtocol":"HTTP/1.1","RequestScheme":"http","RetryAttempts":0,"RouterName":"http-to-https@internal","StartLocal":"2023-11-16T13:34:32.461596348-07:00","StartUTC":"2023-11-16T20:34:32.461596348Z","entryPointName":"http","level":"info","msg":"","time":"2023-11-16T13:34:32-07:00"}

bluepuma77 · November 17, 2023, 1:01pm

{
  "ClientAddr":"<ip>:28398",
  "ClientHost":"<ip>",
  "ClientPort":"28398",
  "ClientUsername":"-",
  "DownstreamContentSize":0,
  "DownstreamStatus":418,
  "Duration":60912,
  "OriginContentSize":0,
  "OriginDuration":0,
  "OriginStatus":0,
  "Overhead":60912,
  "RequestAddr":"www.zerosync.co",
  "RequestContentSize":0,
  "RequestCount":13,
  "RequestHost":"www.zerosync.co",
  "RequestMethod":"GET",
  "RequestPath":"/",
  "RequestPort":"-",
  "RequestProtocol":"HTTP/1.1",
  "RequestScheme":"http",
  "RetryAttempts":0,
  "RouterName":"http-to-https@internal",
  "StartLocal":"2023-11-16T13:34:32.461596348-07:00",
  "StartUTC":"2023-11-16T20:34:32.461596348Z",
  "entryPointName":"http",
  "level":"info",
  "msg":"",
  "time":"2023-11-16T13:34:32-07:00"
}

Router is http-to-https@internal, the OriginStatus from target is not really set, but DownstreamStatus = 418, so it seems this comes directly from Traefik itself.

Compare this to my simple redirect test:

{
  "ClientAddr":"<ip>:51056",
  "ClientHost":"<ip>",
  "ClientPort":"51056",
  "ClientUsername":"-",
  "DownstreamContentSize":17,
  "DownstreamStatus":301,
  "Duration":399262,
  "GzipRatio":0,
  "OriginContentSize":0,
  "OriginDuration":0,
  "OriginStatus":0,
  "Overhead":399262,
  "RequestAddr":"whoami.example.com",
  "RequestContentSize":0,
  "RequestCount":2,
  "RequestHost":"whoami.example.com",
  "RequestMethod":"GET",
  "RequestPath":"/",
  "RequestPort":"-",
  "RequestProtocol":"HTTP/1.1",
  "RequestScheme":"http",
  "RetryAttempts":0,
  "RouterName":"web-to-websecure@internal",
   "StartLocal":"2023-11-17T12:47:56.54538568Z",
  "StartUTC":"2023-11-17T12:47:56.54538568Z",
  "entryPointName":"web",
  "level":"info",
  "msg":"",
  "time":"2023-11-17T12:47:56Z"
}

Still navigating blindly. What is your current Traefik debug log state (as text)? What does Traefik dashboard say, you have it enabled with dashboard: true and insecure: true, so it will be listening on port http://domain:8080/dashboard/. How did you configure your target service?

bluepuma77 · November 17, 2023, 3:33pm

Maybe you start with a simple Traefik example and work you way forward, adding security middleware and Cloudflare later on.

adunne09 · November 17, 2023, 11:04pm

was able to solve it by downgrading to 2.6.7; went back through a fresh configuration and ran into a redirect loop, which I was then able to solve by changing my Cloudflare TLS settings. thanks

bluepuma77 · November 18, 2023, 6:52am

Sorry, "downgrading to 2.6.7" is no "solution", it’s an IT security incident waiting to happen.

system · November 21, 2023, 6:52am

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.