404 Not Found only when using auth@file with Authelia (Used to Work)

Traefik Traefik v3 (latest)
1

Hey y'all,

I have no idea what broke since this all used to work. I'm using Authelia to secure a couple of webapps, and it used to work fine, passing along Remote-User headers and everything. Something must have changed, but I can't narrow it down, and posting here is my last ditch to get it working haha.

I can set up services just fine, but as soon as I add the auth@file middleware the services start returning a 404. I can connect to auth.domain.com just fine, log in, log out, etc, but applying it to a service immediately makes it return a 404 with no further information in any logs or developer tools.

I've attached the relevant snippets below:
Note: Service responds as normal with ipAllowList middleware.

fileConfig.yml Routers

http:
  routers:
    paperless-ngx:
      entryPoints:
        - https
      rule: 'Host(`paperless.domain.com`)'
      service: paperless-ngx
      middlewares:
        - auth@file
        - ipWhiteList@file

fileConfig.yml Service (X for obfuscation)

 services:
    paperless-ngx:
      loadBalancer:
        servers:
          - url: http://192.168.1.X:8000

fileConfig.yml Middlewares

  middlewares:
    # Only Allow Local networks
    ipWhiteList:
      ipWhiteList:
        sourceRange: 
          - 127.0.0.1/32 # localhost
          - 192.168.1.1/24 # LAN Subnet
  
    # Authelia guard
    auth:
      forwardauth:
        address: http://authelia:9091/api/verify?rd=https://auth.domain.com/
        trustForwardHeader: true
        authResponseHeaders:
          - Remote-User
          - Remote-Groups
          - Remote-Name
          - Remote-Email

Does anything here look wrong? I can additionally post my Authelia config, let me know the relevant bits so I can obfuscate and minimise the amount I'm posting.

Thanks so much in advance everyone.

Additional info:
All apps are running in Docker within Unraid. I'm defining the services within the fileConfig to easier keep track/AB test my auth issues. I'm connecting locally for now, but I have some other services exposed to the open internet. If I had to narrow down some changes that broke Authelia, I think I may have made a change during setting up Remote-User headers, but I can't find any differences between my current setup and the example configs I adapted to set up Authelia in the first place.

2

What changed? Was it a major (v2 -> v3) or minor Traefik upgrade? OS upgrade? Unraid upgrade? Docker upgrade?

Enable and check Traefik debug log (doc) and Traefik access log in JSON format (doc).

Share your full Traefik static and dynamic config, and docker-compose.yml if used.

3

Hey, changed all the logs to debug. Traefik responds with

github.com/traefik/traefik/v3/pkg/middlewares/auth/forward.go:177 > Remote error http://authelia:9091/api/verify?rd=http://auth.domain.com/. StatusCode: 404 middlewareName=auth@file middlewareType=ForwardAuth

Which makes me think Authelia isn't responding. On Authelias side, no logs are showing up other than the server is listening. I can't figure out why though. Configs below.

Static:

global:
  checkNewVersion: true
  sendAnonymousUsage: false
serversTransport:
  insecureSkipVerify: true
entryPoints:
  # Not used in apps, but redirect everything from HTTP to HTTPS
  http:
    address: :80
    forwardedHeaders:
      trustedIPs:
        - "127.18.0.1/24"
        - "192.168.1.1/24"
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https
  # HTTPS endpoint, with domain wildcard
  https:
    address: :443
    forwardedHeaders:
      # Reuse list of Cloudflare Trusted IP's above for HTTPS requests
      trustedIPs:
        - "127.18.0.1/24"
        - "192.168.1.1/24"
    http:
      tls:
        # Generate a wildcard domain certificate
        certResolver: letsencrypt
        domains:
          - main: domain.com
            sans:
              - '*.domain.com'
      middlewares:
        - securityHeaders@file
providers:
  providersThrottleDuration: 2s

  # File provider for connecting things that are outside of docker / defining middleware
  file:
    filename: /etc/traefik/fileConfig.yml
    watch: true

  # Docker provider for connecting all apps that are inside of the docker network
  docker:
    watch: true
    network: docker_network    # Add Your Docker Network Name Here
    # Default host rule to containername.domain.example
    defaultRule: "Host(`{{ lower (trimPrefix `/` .Name )}}.domain.com`)"    # Replace with your domain
#    swarmModeRefreshSeconds: 15s
    exposedByDefault: false
# Enable traefik ui
api:
  dashboard: true
  insecure: true
# Log level INFO|DEBUG|ERROR
log:
  level: debug
# Use letsencrypt to generate ssl serficiates
certificatesResolvers:
  letsencrypt:
    acme:
      email: email
      storage: /etc/traefik/acme.json
      dnsChallenge:
        provider: cloudflare
        # Used to make sure the dns challenge is propagated to the rights dns servers
        resolvers:
          - "1.1.1.1:53"
          - "1.0.0.1:53"

Dynamic:

http:
  routers:
    authelia:
      entryPoints:
        - https
      rule: 'Host(`auth.domain.com`)'
      service: authelia
      middlewares:
    whoami:
      entryPoints:
        - https
      rule: 'Host(`whoami.domain.com`)'
      service: whoami
      middlewares:
        - auth@file
    paperless-ngx:
      entryPoints:
        - https
      rule: 'Host(`paperless.domain.com`)'
      service: paperless-ngx
      middlewares:
        - ipAllowList@file
    code-server:
      entryPoints:
        - https
      rule: 'Host(`code.domain.com`)'
      service: code-server
      middlewares:
    homeassistant:
      entryPoints:
        - https
      rule: 'Host(`ha.domain.com`)'
      service: homeassistant
      middlewares:
        - ipAllowList@file
    unraid:
      entryPoints:
        - https
      rule: 'Host(`unraid.domain.com`)'
      service: unraid
      middlewares:
        - ipAllowList@file
  # ## SERVICES ##
  services:
    authelia:
      loadBalancer:
        servers:
          - url: http://192.168.1.9:9091/
    whoami:
      loadBalancer:
        servers:
          - url: http://192.168.1.9:9999
    paperless-ngx:
      loadBalancer:
        servers:
          - url: http://192.168.1.9:8000
    code-server:
      loadBalancer:
        servers:
          - url: http://192.168.1.9:8672
    homeassistant:
      loadBalancer:
        servers:
          - url: http://192.168.1.2:8123/
    unraid:
      loadBalancer:
        servers:
          - url: https://192.168.1.9:8443/

  ## MIDDLEWARES ##
  middlewares:
    # Only Allow Local networks
    ipAllowList:
      ipAllowList:
        sourceRange: 
          - 127.18.0.1/32 # localhost
          - 192.168.1.1/24 # LAN Subnet
    auth:
      forwardauth:
        address: http://authelia:9091/api/verify?rd=http://auth.domain.com/ # replace auth with your authelia container name
        trustForwardHeader: true
        authResponseHeaders:
          - Remote-User
          - Remote-Groups
          - Remote-Name
          - Remote-Email
    # Security headers
    securityHeaders:
      headers:
        customResponseHeaders:
          X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex"
          X-Forwarded-Proto: "https"
          server: ""
        customRequestHeaders:
          X-Forwarded-Proto: "https"
        sslProxyHeaders:
          X-Forwarded-Proto: "https"
        referrerPolicy: "same-origin"
        hostsProxyHeaders:
          - "X-Forwarded-Host"
        contentTypeNosniff: true
        browserXssFilter: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsSeconds: 63072000
        stsPreload: true
 
# Only use secure ciphers - https://ssl-config.mozilla.org/#server=traefik&version=2.6.0&config=intermediate&guideline=5.6              
tls:
  options:
    default:
      minVersion: VersionTLS12
      cipherSuites:
        - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305

I've got everything in the dynamic just so I can test things easier, I used to have these set up as docker labels. Anything unusual?

4

Managed to fix this one on my own, genuinely took hours since I struggled to find any indication on any Authelia documentation anywhere. (Turns out to not be a Traefik issue at all)

If Authelia is using Authz as a provider, then the usual /auth/verify won't work anymore. Your ingress route has to be as follows:

   auth:
      forwardAuth:
        address: "http://authelia:9091/api/authz/forward-auth?rd=https://auth.domain.com/"

Good luck future troubleshooters.

1 Like