ERR_ECH_FALLBACK_CERTIFICATE_INVALID errors

I am currently experiencing a very specific issue with one of my services and Traefik. On my phone, every time I restart my browser and try to access my Nextcloud locally, I am getting an ERR_ECH_FALLBACK_CERTIFICATE_INVALID error. If I access any other service on the same host with a different subdomain and then go back to Nextcloud, it works again, but only until I restart the browser again. It doesn't seem to affect any of my other devices. Additionally, the access log or any other log doesn't actually show that my phone even attempts to connect. I also attached my traefik.yml, nextcloud.yaml and plugin configuration files here as well as my compose file, if it is a configuration issue.

compose.yaml

services:
  traefik:
    container_name: traefik
    image: traefik:v3.1
    # Enables the web UI and tells Traefik to listen to docker
    ports:
      # The HTTP port
      - 192.168.178.70:80:80
      - 192.168.178.70:443:443
    restart: always
    volumes:
      # So that Traefik can listen to the Docker events
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./traefik.yml:/etc/traefik/traefik.yml:ro
      - ./dynamic-conf:/etc/traefik/dynamic-conf
      - ./certs:/certs
      - traefik-logs:/var/log/traefik
      - ./captcha.html:/captcha.html:ro
      - ./ban.html:/ban.html
    env_file: .env
    environment:
      - DUCKDNS_TOKEN=${DUCKDNS_TOKEN}
      - CF_API_EMAIL=${CF_API_EMAIL}
      - CF_DNS_API_TOKEN=${CF_DNS_API_TOKEN}
      - TZ=Europe/Berlin
    networks:
      - proxy
    extra_hosts:
      - host.docker.internal:host-gateway
    labels:
      - diun.enable=true
      - traefik.enable=true
      - traefik.docker.network=proxy
      - traefik.http.routers.traefik-dashboard.rule=Host(`traefik.<redacted>`)
      - traefik.http.routers.traefik-dashboard.service=api@internal
      - traefik.http.middlewares.traefik-auth.basicauth.users=${DASHBOARD_CREDENTIALS}
      - traefik.http.routers.traefik-dashboard.middlewares=traefik-auth
networks:
  proxy:
    external: true
volumes:
  traefik-logs: null

traefik.yml

api:
  dashboard: true

certificatesResolvers:
  letsencrypt:
    acme:
      email: <redacted>
#      caServer: https://acme-staging-v02.api.letsencrypt.org/directory
      storage: /certs/acme/alphacraft-cloudflare.json
      dnschallenge:
        provider: cloudflare
        delayBeforeCheck: 60
#        disablePropagationCheck: false

serversTransport:
  insecureSkipVerify: true

entryPoints:
  http:
    # Listen on port 80 for incoming http requests
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https
          permanent: true
  https:
    # Listen on port 443 for incoming https requests
    address: ":443"
    AsDefault: true
    http:
      middlewares:
        - cloudflarewarp@file
        - crowdsec@file
      encodeQuerySemicolons: true
      tls:
        certResolver: letsencrypt
        domains:
          - main: "<redacted>"
            sans: "*.<redacted>"
    http3:
      advertisedPort: 443
    forwardedHeaders:
      trustedIPs:
        - "192.168.178.1/24"
        - "192.168.32.1/24"
        - "172.26.0.2"
    proxyProtocol:
      trustedIPs:
        - "192.168.178.1/24"
        - "192.168.32.1/24"

providers:
  file:
    directory: /etc/traefik/dynamic-conf
    watch: true
  docker:
    network: proxy

log:
  level: "INFO"
  filePath: "/var/log/traefik/traefik.log"

accessLog:
  filePath: "/var/log/traefik/access.log"

experimental:
  plugins:
    bouncer:
      moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin
      version: v1.3.3
    cloudflarewarp:
      modulename: github.com/BetterCorp/cloudflarewarp
      version: v1.3.3

nextcloud.yaml

http:
  routers:
    nextcloud:
      rule: "Host(`<redacted>`)"
      entrypoints:
        - "https"
      service: nextcloud
      middlewares:
        - nextcloud-chain
#      tls: {}
#        certresolver: "letsencrypt"

  services:
    nextcloud:
      loadBalancer:
        servers:
          - url: "http://192.168.178.70:11000" # Use the host's IP address if Traefik runs outside the host network

  middlewares:
    nextcloud-secure-headers:
      headers:
        hostsProxyHeaders:
          - "X-Forwarded-Host"
        referrerPolicy: "same-origin"

#    https-redirect:
#      redirectscheme:
#        scheme: https

    nextcloud-chain:
      chain:
        middlewares:
          # - ... (e.g. rate limiting middleware)
#          - https-redirect
          - nextcloud-secure-headers

crowdsec.yaml

http:
  middlewares:
    crowdsec:
      plugin:
        bouncer:
          enabled: true
          logLevel: INFO
          # updateIntervalSeconds: 60
          # updateMaxFailure: 0
          defaultDecisionSeconds: 60
          httpTimeoutSeconds: 10
          crowdsecMode: live
          crowdsecAppsecEnabled: true
          crowdsecAppsecHost: crowdsec:7422
          crowdsecAppsecFailureBlock: true
          crowdsecAppsecUnreachableBlock: true
          crowdsecLapiKey: <redacted>
#          crowdsecLapiKeyFile: /etc/traefik/cs-privateKey-foo
          crowdsecLapiHost: crowdsec:8080
          crowdsecLapiScheme: http
          # crowdsecLapiTLSInsecureVerify: false
          # crowdsecCapiMachineId: login
          # crowdsecCapiPassword: password
          # crowdsecCapiScenarios:
          #   - crowdsecurity/http-path-traversal-probing
          #   - crowdsecurity/http-xss-probing
          #   - crowdsecurity/http-generic-bf
          forwardedHeadersTrustedIPs:
            - 10.0.10.23/32
            - 10.0.20.0/24
            - 192.168.178.1/24
            - 192.168.32.1/24
          clientTrustedIPs:
            - 192.168.1.0/24
            - 192.168.32.1/24
            - 192.168.178.1/24
          forwardedHeadersCustomName: X-Real-IP
          redisCacheEnabled: false
          captchaProvider: hcaptcha
          captchaSiteKey: <redacted>
          # captchaSecretKey: FIXME
          captchaGracePeriodSeconds: 1800
          captchaHTMLFilePath: /captcha.html
          banHTMLFilePath: /ban.html

cloudflarewarp.yaml

http:
  middlewares:
    cloudflarewarp:
      plugin:
        cloudflarewarp:
          disableDefault: true
          trustip:
            - 192.168.32.1/24

I hope that I haven't missed any important information and that someone is able to help me with the issue. I haven't been able to figure anything out, even after hours of research. Thanks for any help in advance!

Two more pieces of info: I tried rebooting my phone, but that definitely wasn't the culprit. This issue also doesn't affect access via Cloudflare at all. So this seems to be a local issue with Traefik, which makes my phone not be able to properly access Nextcloud unless I access a different service first (at each browser restart). It could also be some other issue with my phone, but it really looks like a Traefik problem.

You are sure it’s about requesting a different site, not about timing?

Yes. This config option only applies to the dns challenge for requesting a certificate from letsencrypt. It waits for a minute to let the DNS entries propagate, so that it doesn't risk hitting any letsencrypt rate-limits. The certificate already exists on the server and is valid by itself. Other services have no issues on my phone.

Is there a reason you used the UDP tag?

For debug I would probably change the image to traefik/whoami, keep all settings, to see if it behaves differently.

"ECH" or "Encrypted client hello" is related to http 1.3 as far as I know and there was something with UDP there. I can try to edit the dynamic config file for nextcloud to point it to whoami. I'll test that very soon.

I tried it with whoami now. Initially, it seemed to work, but it now also has the same issue. Same config, except for the port. Still the same issue. It's in the nextcloud.yaml, but it appears to only affect my phone.

It's really weird. It only appears to affect this one single device and even only this one browser on it (brave). AOSP browser has no issues. But resetting Brave completely also hasn't helped with this issue...

Okay, it isn't actually related to my browser. It didn't affect AOSP browser, but it did affect Firefox on the same device. It's just the phone or maybe some misconfiguration on Traefik which triggers my phone only somehow... I don't know what's going on.

Have you tried narrowing it down? Like remove additional setting bit by bit?

It seems you use CloudFlare, are you sure it is supporting http3?

Also http3 is using UDP, you would need to open the ports additionally in UDP mode in compose.

I tried removing some settings one by one, like the manual tls declaration in the dynamic config file, as I have already set it up globally. Cloudflare itself sorry ECH but the issue only affects local access using my phone. I think the latter is actually the culprit. I'll try to manually forward the port for UDP in the compose file. Should it only be port 443, or also port 80?

It doesn't seem to the a UDP problem either. Adding the UDP port bindings doesn't seem to solve the issue. I'm going to try other settings.

Try without http3 enabled.

I'll try that. What's the benefit of using it anyway?

I tried with http 3 disabled, and even without the nextcloud chain Middleware, but to no avail. What could be going on? It worked fine recently.

I tested out, whether binding the subdomain to an entirely different service would help, but it doesn't. So it doesn't really look like a Nextcloud or a Traefik problem. My phone could be doing something weird, but I don't know how. Or Traefik has somehow flagged my phone's IP, or something like that might be going on. I don't know how to check all of these different things.

1 Like

I have the same issue with Chrome on PC

This is a very weird issue. It only affects my phone, but system-wide. I tested across browsers, in Brave, Firefox and Chrome. All of them have the same issue. I want to note again, that I both use Cloudflare tunnels and Traefik, just that in my home network, I directly access Nextcloud via Traefik, skipping Cloudflare. It is something with an invalid fallback certificate and occurs in the encrypted client hello. This is all I know and I have no real clue as to what might be causing this and what I could attempt to fix it. Any help is very appreciated.