Dashboard & Let's Encrypt Certificates not working no matter what I tried

luckylinux · January 7, 2024, 10:45am

I am running Traefik (latest version) using Podman 4.7.2 & Podman-Compose 1.0.6.

Lots of things tried (and left as commented), but nothing really works.
Initially it seemed like the certificate was valid, but that was just because I was behind Cloudflare DNS Server with Proxy Enabled. I switched to DNS only now, so there is one less thing between me & the server to debug.

version: "3.8"
networks:
  podman:
    external: true
services:
  traefik:
    image: traefik:latest
    pull_policy: always
    restart: unless-stopped
    container_name: traefik
    command:
      - "--log.level=DEBUG"
      - "--log.filePath=/log/traefik.log"
      - "--accesslog=true"
      - "--accesslog.filePath=/log/access.log"
      - "--api.insecure=false" # production = false , development = true
      - "--api.dashboard=true"
#####      - "--entrypoints.web.address=:8000"
#####      - "--entrypoints.websecure.address=:8443"
#      - "--entrypoints.admin.address=:8100"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
#####      - "--entrypoints.websecure.asDefault=true"
      - "--entrypoints.websecure.http.tls.certresolver=lets-encrypt"
      - "--certificatesresolvers.lets-encrypt.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
      - "--certificatesresolvers.lets-encrypt.acme.email=MYEMAIL@MYTLD"
      - "--certificatesresolvers.lets-encrypt.acme.storage=/letsencrypt/acme.json"
      - "--certificatesresolvers.lets-encrypt.acme.httpchallenge.entrypoint=web"
      - "--certificatesresolvers.lets-encrypt.acme.tlschallenge=false"
#####      - "--entrypoints.http.address=:80"
#####      - "--entrypoints.http.address=:8000"
#      - "--entrypoints.http.http.redirections.entryPoint.to=websecure"
#      - "--entrypoints.http.http.redirections.entryPoint.scheme=https"
#####      - "--entrypoints.https.address=:443"
#####      - "--entrypoints.https.address=:8443"
      - "--providers.docker=true"
      - "--serversTransport.insecureSkipVerify=true"
      - "--global.sendAnonymousUsage=false"
    labels:
      - "traefik.enable=true"
#      - "traefik.docker.network=web"
#      - "traefik.http.routers.api.rule=Host(`traefik.MYDOMAIN.TLD`)"
      - "traefik.http.routers.api.entrypoints=websecure"
      - "traefik.http.routers.api.rule=Host(`traefik.MYDOMAIN.TLD`) && PathPrefix(`/api`)"
#      - "traefik.http.routers.api.rule=PathPrefix(`/api`)"
      - "traefik.http.routers.api.service=api@internal"
      - "traefik.http.routers.api.middlewares=secured@file"
      - "traefik.http.routers.dashboard.entrypoint=websecure"
      - "traefik.http.routers.dashboard.rule=Host(`traefik.MYDOMAIN.TLD`) && PathPrefix(`/dashboard`)"
#      - "traefik.http.routers.dashboard.rule=PathPrefix(`/dashboard`)"
      - "traefik.http.routers.dashboard.service=api@internal"
      - "traefik.http.routers.dashboard.middlewares=auth"
#      - "traefik.http.routers.dashboard.middlewares=dashboard-https"
######      - "traefik.http.middlewares.dashboard-https.redirectscheme.scheme=https"
#      - "traefik.http.routers.dashboard-secure.entrypoints=websecure"
######      - "traefik.http.routers.dashboard-secure.rule=Host(`traefik.MYDOMAIN.TLD`) && PathPrefix(`/api`, `/dashboard`)"
###      - "traefik.http.routers.dashboard-secure.rule=Host(`traefik.MYDOMAIN.TLD`) && PathPrefix(`/dashboard`)"
###      - "traefik.http.routers.dashboard-secure.tls=true"
###      - "traefik.http.routers.dashboard-secure.service=api@internal"
###      - "traefik.http.routers.dashboard-secure.tls.certresolver=letls"
#      - "traefik.http.middlewares.auth.basicauth.usersfile=/config/users"
    networks:
      - podman
    ports:
      - 80:80
      - 443:443
#      - 8100:8100
#      - 8000:8000 # web UI (enabled with --api.insecure=true)
#      - 8443:8443 # web UI
    volumes:
      - ~/data/traefik/letsencrypt:/letsencrypt
      - ~/config/traefik:/config
      - ~/certificates/traefik:/certificates
      - ~/log/traefik:/log
      - /run/user/1001/podman/podman.sock:/var/run/docker.sock:z
#   - /var/run/docker.sock:/var/run/docker.sock:ro
#   - --providers.docker

Concerning the SSL/TLS Certificate, I am seeing this in my logs:

time="2024-01-07T10:16:23Z" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
time="2024-01-07T10:16:23Z" level=debug msg="Added outgoing tracing middleware acme-http@internal" entryPointName=web routerName=acme-http@internal middlewareName=tracing middlewareType=TracingForwarder
time="2024-01-07T10:16:23Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=web middlewareName=traefik-internal-recovery
time="2024-01-07T10:16:30Z" level=debug msg="Serving default certificate for request: \"traefik.MYDOMAIN.TLD\""
time="2024-01-07T10:16:30Z" level=debug msg="http: TLS handshake error from 10.88.0.46:34358: remote error: tls: bad certificate"
time="2024-01-07T10:16:31Z" level=debug msg="Serving default certificate for request: \"traefik.MYDOMAIN.TLD\""
time="2024-01-07T10:16:31Z" level=debug msg="http: TLS handshake error from 10.88.0.46:34374: remote error: tls: bad certificate"
time="2024-01-07T10:17:35Z" level=debug msg="Serving default certificate for request: \"traefik.MYDOMAIN.TLD\""
time="2024-01-07T10:17:35Z" level=debug msg="http: TLS handshake error from 10.88.0.46:47964: remote error: tls: bad certificate"
time="2024-01-07T10:17:38Z" level=debug msg="Serving default certificate for request: \"traefik.MYDOMAIN.TLD\""

The acme.json file was never even touched. It's currently empty and set with the correct permissions:

  File: /home/podman/data/traefik/letsencrypt/acme.json
  Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
Device: 8,1	Inode: 8839        Links: 1
Access: (0600/-rw-------)  Uid: ( 1001/  podman)   Gid: ( 1001/  podman)
Access: 2024-01-06 20:49:56.547451180 +0100
Modify: 2024-01-06 20:10:04.231409608 +0100
Change: 2024-01-06 20:15:54.115651694 +0100
 Birth: 2024-01-06 20:10:04.231409608 +0100

Whereas concerning the Dashboard I see:

10.88.0.44 - - [07/Jan/2024:10:09:20 +0000] "GET /.well-known/acme-challenge/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX HTTP/1.1" 404 19 "-" "-" 7 "-" "-" 0ms
10.88.0.44 - - [07/Jan/2024:10:09:52 +0000] "GET / HTTP/1.1" 404 19 "-" "-" 8 "-" "-" 0ms
10.88.0.44 - - [07/Jan/2024:10:13:20 +0000] "GET / HTTP/1.1" 404 19 "-" "-" 9 "-" "-" 0ms
10.88.0.45 - - [07/Jan/2024:10:16:07 +0000] "GET /test/.git/config HTTP/1.1" 404 19 "-" "-" 1 "-" "-" 0ms
10.88.0.46 - - [07/Jan/2024:10:17:38 +0000] "GET /dashboard HTTP/2.0" 404 19 "-" "-" 1 "-" "-" 1ms
10.88.0.46 - - [07/Jan/2024:10:17:38 +0000] "GET /favicon.ico HTTP/2.0" 404 19 "-" "-" 2 "-" "-" 0ms
10.88.0.46 - - [07/Jan/2024:10:31:06 +0000] "GET /dashboard HTTP/2.0" 404 19 "-" "-" 3 "-" "-" 0ms
10.88.0.46 - - [07/Jan/2024:10:31:06 +0000] "GET /favicon.ico HTTP/2.0" 404 19 "-" "-" 4 "-" "-" 0ms
10.88.0.46 - - [07/Jan/2024:10:31:07 +0000] "GET /dashboard HTTP/2.0" 404 19 "-" "-" 5 "-" "-" 0ms
10.88.0.46 - - [07/Jan/2024:10:31:07 +0000] "GET /favicon.ico HTTP/2.0" 404 19 "-" "-" 6 "-" "-" 0ms

Not sure what is trying to access /test/.git but otherwise it seems letsencrypt is trying to perform the acme-challange response and not finding the file (404). Similarly, I cannot access the Dashboard.

I tried both --certificatesresolvers.lets-encrypt.acme.tlschallenge=true and --certificatesresolvers.lets-encrypt.acme.tlschallenge=false but couldn't observe any difference.

Any idea what is going on ?

From what I could see, currently internal (container) DNS resolution is not working, which might explain some of the issues. I am using the default "podman" network bridge which, for compatibility with Docker, has dns_enabled = false. Not sure if this could be the issue.

But that doesn't explain all of the 404 / Not found messages by itself.

podman network inspect podman
[
     {
          "name": "podman",
          "id": "2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9",
          "driver": "bridge",
          "network_interface": "podman0",
          "created": "2024-01-07T11:43:40.240233866+01:00",
          "subnets": [
               {
                    "subnet": "10.88.0.0/16",
                    "gateway": "10.88.0.1"
               }
          ],
          "ipv6_enabled": false,
          "internal": false,
          "dns_enabled": false,
          "ipam_options": {
               "driver": "host-local"
          }
     }
]

luckylinux · January 7, 2024, 7:36pm

Actually I kind-of managed to get it running.

Trying to introduce crowdsec breaks it again though ...

The incriminating line appears to be
traefik.http.routers.dashboard.entrypoint=websecure

Removing that seems to fix the issue.

Updated compose.yml file - crowdsec-bouncer@docker has to be disabled otherwise nothing currently works. That's a separate issue ...

version: '3.9'

networks:
  traefik:
    external: true

services:
  traefik:
    image: traefik:latest
    hostname: traefik.XXXXXXXXX
    domainname: XXXXXXXXX
    restart: unless-stopped
    container_name: traefik
    ports:
      - 80:80
      - 443:443
#      - 8080:8080
#      - 8443:8443
    networks:
      - traefik
    volumes:
      - /run/user/1001/podman/podman.sock:/var/run/docker.sock:ro
      - ~/data/traefik/letsencrypt:/letsencrypt
      - ~/config/traefik:/config
      - ~/certificates/traefik:/certificates
      - ~/log/traefik:/log
    command:
      ## Logging
      # Server Log
#      - "--log.level=DEBUG"
      - "--log.level=INFO"
      - "--log.filePath=/log/traefik.log"

      # Error Log
      - "--accesslog=true"
      - "--accesslog.filePath=/log/access.log"

      ## Dashboard & API
      - "--api"
      - "--api.insecure=false" # production = false , development = true
      - "--api.dashboard=true"

      ## EntryPoints
      # Unsecure Connection - Redirect to Secure
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
      - "--entryPoints.web.http.redirections.entrypoint.scheme=https"
      - "--entrypoints.web.http.redirections.entrypoint.permanent=true"

      # Secure Connection
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.websecure.http.tls=true"
      - "--entrypoints.websecure.http.tls.certresolver=letsencrypt"

      ## Letsencrypt Configuration
#      - "--certificatesresolvers.lets-encrypt.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory" # For testing only
      - "--certificatesresolvers.letsencrypt.acme.caserver=https://acme-v02.api.letsencrypt.org/directory" # Production - Rated limited !!!
      - "--certificatesresolvers.letsencrypt.acme.email=XXXXXXXXXXXXX"
      - "--certificatesresolvers.letsencrypt.acme.tlschallenge=true"
      - "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json"
      - "--certificatesresolvers.letsencrypt.acme.httpchallenge=true"                 # Not sure if needed
      - "--certificatesresolvers.letsencrypt.acme.httpchallenge.entrypoint=websecure" # Not sure if needed

      ## Docker / Podman Intergration
      - "--providers.docker=true"
      - "--providers.docker.exposedByDefault=false"
      - "--providers.docker.watch=false"
      - "--providers.docker.swarmMode=false"
      - "--providers.docker.endpoint=unix:///var/run/docker.sock"
#      - "--providers.docker.network=proxy"          # From Tutorial
#      - "--providers.docker.exposedByDefault=false" # From Tutorial

      # Crowdsec Integration
      #- "--providers.file=true"
      #- "--providers.file.filename=/config/config.yml"

      ## Other
      # ...
      - "--serversTransport.insecureSkipVerify=true"

      # No Telemetry
      - "--global.sendAnonymousUsage=false"

    labels:
      # Enable Traefik
      - traefik.enable=true

      # Dashboard
#      - "traefik.http.routers.dashboard.entrypoint=websecure" # !! If enabled, this line causes a 404 page not found for the dashboard !!
#      - "traefik.http.routers.dashboard.rule=Host(`traefik.XXXXXXXXX`)"
      - "traefik.http.routers.dashboard.rule=Host(`traefik.XXXXXXXXX`) && PathPrefix(`/api` , `/dashboard`)"
      - "traefik.http.routers.dashboard.service=api@internal"
      - "traefik.http.routers.dashboard.middlewares=authtraefik , crowdsec-bouncer@docker"

      # Authentication for Dashboard access
#      - "traefik.http.middlewares.authtraefik.basicauth.users=test:$$apr1$$H6uskkkW$$IgXLP6ewTrSuBkTrqE8wj/"
      - "traefik.http.middlewares.authtraefik.basicauth.usersfile=/config/users"

      # Use crowdsec
      # Create bouncer middleware
      - "traefik.http.middlewares.crowdsec-bouncer.forwardauth.address=http://bouncer-traefik:8082/api/v1/forwardAuth"
      - "traefik.http.middlewares.crowdsec-bouncer.forwardauth.trustForwardHeader=true"

bluepuma77 · January 7, 2024, 9:43pm

Check and compare with simple Traefik example (link).

Are you sure podman.sock is fully compatible to docker.sock for providers.docker to work?

luckylinux · January 8, 2024, 6:03am

That's actually what I did. Start with your file, rename "proxy" network to "traefik", change email address & domain name and keep copying one-by-one my previous configuration lines into your example file until it stopped working.

Every time I issued a

podman traefik stop
podman-compose up -d

As I said, that's how I found that the line that breaks EVERYTHING was

traefik.http.routers.dashboard.entrypoint=websecure

I presume so. The issue seems rather to be that crowdsec has some SERIOUS parsing issue. I tried to open a post on their forum, but the AKISMET spam bot put that into moderation as well. Guess it's not only this forum where the spam bot is very efficient in flagging user content .

Basically the errors are like

time="07-01-2024 19:14:17" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:14:17Z\" level=info msg=\"Server stopped\""
time="07-01-2024 19:14:17" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:14:17" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:14:17Z\" level=info msg=\"Shutting down\""
time="07-01-2024 19:14:17" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:14:18" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:14:18Z\" level=info msg=\"Traefik version 2.10.7 built on 2023-12-06T15:54:59Z\""
time="07-01-2024 19:14:18" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:14:18" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:14:18Z\" level=info msg=\"\\nStats collection is disabled.\\nHelp us improve Traefik by turning this feature on :)\\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\\n\""
time="07-01-2024 19:14:18" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:14:18" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:14:18Z\" level=info msg=\"Starting provider aggregator aggregator.ProviderAggregator\""
time="07-01-2024 19:14:18" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:14:18" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:14:18Z\" level=info msg=\"Starting provider *traefik.Provider\""
time="07-01-2024 19:14:18" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:14:18" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:14:18Z\" level=info msg=\"Starting provider *docker.Provider\""
time="07-01-2024 19:14:18" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:14:18" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:14:18Z\" level=info msg=\"Starting provider *acme.ChallengeTLSALPN\""
time="07-01-2024 19:14:18" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:14:18" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:14:18Z\" level=info msg=\"Starting provider *acme.Provider\""
time="07-01-2024 19:14:18" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:14:18" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:14:18Z\" level=info msg=\"Testing certificate renew...\" providerName=letsencrypt.acme ACME CA=\"https://acme-v02.api.letsencrypt.org/directory\""
time="07-01-2024 19:14:18" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:14:18" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:14:18Z\" level=warning msg=\"No domain found in rule PathPrefix(`/.well-known/acme-challenge/`), the TLS options applied for this router will depend on the SNI of each request\" entryPointName=websecure routerName=acme-http@internal"
time="07-01-2024 19:14:18" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:14:19" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:14:19Z\" level=warning msg=\"No domain found in rule PathPrefix(`/.well-known/acme-challenge/`), the TLS options applied for this router will depend on the SNI of each request\" routerName=acme-http@internal entryPointName=websecure"
time="07-01-2024 19:14:19" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:14:19" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:14:19Z\" level=warning msg=\"No domain found in rule PathPrefix(`/.well-known/acme-challenge/`), the TLS options applied for this router will depend on the SNI of each request\" entryPointName=websecure routerName=acme-http@internal"
time="07-01-2024 19:14:19" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:14:48" level=info msg="127.0.0.1 - [Sun, 07 Jan 2024 19:14:48 UTC] \"GET /v1/heartbeat HTTP/1.1 200 6.793478ms \"crowdsec/v1.5.5-d2d788c5dc0a9e387635276623c6781774a9dfd4\" \""
time="07-01-2024 19:15:48" level=info msg="127.0.0.1 - [Sun, 07 Jan 2024 19:15:48 UTC] \"GET /v1/heartbeat HTTP/1.1 200 8.296261ms \"crowdsec/v1.5.5-d2d788c5dc0a9e387635276623c6781774a9dfd4\" \""
time="07-01-2024 19:16:48" level=info msg="127.0.0.1 - [Sun, 07 Jan 2024 19:16:48 UTC] \"GET /v1/heartbeat HTTP/1.1 200 6.491173ms \"crowdsec/v1.5.5-d2d788c5dc0a9e387635276623c6781774a9dfd4\" \""
time="07-01-2024 19:17:48" level=info msg="127.0.0.1 - [Sun, 07 Jan 2024 19:17:48 UTC] \"GET /v1/heartbeat HTTP/1.1 200 10.882438ms \"crowdsec/v1.5.5-d2d788c5dc0a9e387635276623c6781774a9dfd4\" \""
time="07-01-2024 19:17:53" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:17:53Z\" level=info msg=\"I have to go...\""
time="07-01-2024 19:17:53" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:17:53" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:17:53Z\" level=info msg=\"Stopping server gracefully\""
time="07-01-2024 19:17:53" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:17:53" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:17:53Z\" level=error msg=\"accept tcp [::]:443: use of closed network connection\" entryPointName=websecure"
time="07-01-2024 19:17:53" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:17:53" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:17:53Z\" level=error msg=\"Error while starting server: accept tcp [::]:443: use of closed network connection\" entryPointName=websecure"
time="07-01-2024 19:17:53" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:17:53" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:17:53Z\" level=error msg=\"accept tcp [::]:80: use of closed network connection\" entryPointName=web"
time="07-01-2024 19:17:53" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:17:53" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:17:53Z\" level=error msg=\"Error while starting server: accept tcp [::]:80: use of closed network connection\" entryPointName=web"
time="07-01-2024 19:17:53" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:17:53" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:17:53Z\" level=info msg=\"Server stopped\""
time="07-01-2024 19:17:53" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:17:53" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:17:53Z\" level=info msg=\"Shutting down\""
time="07-01-2024 19:17:53" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:17:58" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:17:58Z\" level=info msg=\"Traefik version 2.10.7 built on 2023-12-06T15:54:59Z\""
time="07-01-2024 19:17:58" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:17:58" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:17:58Z\" level=info msg=\"\\nStats collection is disabled.\\nHelp us improve Traefik by turning this feature on :)\\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\\n\""
time="07-01-2024 19:17:58" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:17:58" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:17:58Z\" level=info msg=\"Starting provider aggregator aggregator.ProviderAggregator\""
time="07-01-2024 19:17:58" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:17:58" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:17:58Z\" level=info msg=\"Starting provider *traefik.Provider\""
time="07-01-2024 19:17:58" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:17:58" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:17:58Z\" level=info msg=\"Starting provider *docker.Provider\""
time="07-01-2024 19:17:58" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:17:58" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:17:58Z\" level=info msg=\"Starting provider *acme.ChallengeTLSALPN\""
time="07-01-2024 19:17:58" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:17:58" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:17:58Z\" level=info msg=\"Starting provider *acme.Provider\""
time="07-01-2024 19:17:58" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:17:58" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:17:58Z\" level=info msg=\"Testing certificate renew...\" providerName=letsencrypt.acme ACME CA=\"https://acme-v02.api.letsencrypt.org/directory\""
time="07-01-2024 19:17:58" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:17:58" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:17:58Z\" level=warning msg=\"No domain found in rule PathPrefix(`/.well-known/acme-challenge/`), the TLS options applied for this router will depend on the SNI of each request\" entryPointName=websecure routerName=acme-http@internal"
time="07-01-2024 19:17:58" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:17:58" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:17:58Z\" level=warning msg=\"No domain found in rule PathPrefix(`/.well-known/acme-challenge/`), the TLS options applied for this router will depend on the SNI of each request\" entryPointName=websecure routerName=acme-http@internal"
time="07-01-2024 19:17:58" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse
time="07-01-2024 19:17:59" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-01-07T19:17:59Z\" level=warning msg=\"No domain found in rule PathPrefix(`/.well-known/acme-challenge/`), the TLS options applied for this router will depend on the SNI of each request\" routerName=acme-http@internal entryPointName=websecure"
time="07-01-2024 19:17:59" level=warning msg="failed to run filter : invalid character 'i' in literal true (expecting 'r') (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=floral-river name=child-crowdsecurity/traefik-logs stage=s01-parse

youngt2 · January 10, 2024, 2:11pm

Which version of the traefik bouncer are you using? You could try the plugin version that has been updated more recently: GitHub - maxlerebourg/crowdsec-bouncer-traefik-plugin: Traefik plugin to apply crowdsec decisions from local API

You could also try the crowdsec discord - the team there is very helpful: CrowdSec

I see both the TLS and HTTP challenge listed in your lets encrypt. I think it should only be one? Also if you are using http challenge, the documentation says it need to be reachable via port 80, not 443. Traefik Let's Encrypt Documentation - Traefik

Topic		Replies	Views
Unable to get HTTPS to work at all Traefik v3 (latest) docker , letsencrypt-acme	2	499	March 2, 2024
Securing Traefik Dashboard unsuccesfull Traefik v3 (latest) docker , letsencrypt-acme	3	25	April 27, 2025
Lets encrypt certificate doesn't work Traefik v2 letsencrypt-acme	2	704	March 25, 2022
HTTPS let's encrypt not working Traefik v2 docker , letsencrypt-acme	2	4387	November 15, 2019
ERR Router uses a nonexistent certificate resolver Traefik v3 (latest) file , letsencrypt-acme	0	85	February 6, 2025

Dashboard & Let's Encrypt Certificates not working no matter what I tried

Related topics