Hello,
I am struggling to figure out what is incorrect in my setup.
The closest I came to a solution was finding this issue, which was closed without a solution as far as I could see.
I have a docker compose of traefik + crowdsec + bouncer + cloudflared that looks as follows
services:
traefik:
image: traefik:latest
container_name: traefik
restart: unless-stopped
security_opt:
- no-new-privileges:true
networks:
- proxy
- cf-proxy
ports:
- 80:80
- 443:443
- 1883:1883
- 8883:8883
environment:
- CF_API_EMAIL=<email>
- CF_DNS_API_TOKEN=<api key>
volumes:
- /etc/localtime:/etc/localtime:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
- ./docker-data/traefik/config:/config:rw
- ./docker-data/traefik/traefik.yml:/traefik.yml:ro
- ./docker-data/traefik/logs:/var/log/traefik
labels:
- "traefik.enable=true"
- "traefik.http.routers.traefik-secure.entrypoints=https"
- "traefik.http.routers.traefik-secure.rule=Host(`traefik.<domain>`)"
- "traefik.http.routers.traefik-secure.tls=true"
- "traefik.http.routers.traefik-secure.tls.certresolver=cloudflare"
- "traefik.http.routers.traefik-secure.tls.domains[0].main=<domain>"
- "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.<domain>"
- "traefik.http.routers.traefik-secure.service=api@internal"
cloudflared:
image: cloudflare/cloudflared:latest
container_name: cloudflared
restart: unless-stopped
networks:
- cf-proxy
command: tunnel --protocol http2 --no-autoupdate run --token <token>
crowdsec:
image: crowdsecurity/crowdsec:latest
logging:
options:
max-size: "10m"
max-file: "3"
container_name: crowdsec
environment:
GID: "${GID-1000}"
COLLECTIONS: "crowdsecurity/traefik crowdsecurity/http-cve crowdsecurity/base-http-scenarios"
depends_on: #uncomment if running traefik in the same compose file
- 'traefik'
volumes:
- ./docker-data/crowdsec/crowdsec-db:/var/lib/crowdsec/data/
- ./docker-data/crowdsec/crowdsec-config:/etc/crowdsec/
- ./docker-data/traefik/logs:/var/log/traefik/:ro
restart: unless-stopped
networks:
- proxy
labels:
- "traefik.enable=true"
- "traefik.http.routers.crowdsec-secure.entrypoints=https"
- "traefik.http.routers.crowdsec-secure.rule=Host(`crowdsec.<domain>`)"
- "traefik.http.routers.crowdsec-secure.tls=true"
- "traefik.http.routers.crowdsec-secure.service=crowdsec"
- "traefik.http.services.crowdsec.loadbalancer.server.port=8080"
- "traefik.http.services.crowdsec.loadbalancer.server.scheme=http"
- "traefik.docker.network=proxy"
bouncer-traefik:
image: docker.io/fbonalair/traefik-crowdsec-bouncer:latest
logging:
options:
max-size: "10m"
max-file: "3"
container_name: bouncer-traefik
environment:
CROWDSEC_BOUNCER_API_KEY: <api key>
CROWDSEC_AGENT_HOST: crowdsec:8080
networks:
- proxy # same network as traefik + crowdsec
depends_on:
- crowdsec
restart: unless-stopped
networks:
proxy:
name: proxy
cf-proxy:
name: cf-proxy
I have some services I host internally only and access via VPN. But some services I want to be reachable without needing VPN. Hence the cloudflared.
A little while ago, I am not sure when it happened exactly I started getting cloudflare error 1000 errors on some of my services. Internally these services resolve just fine. Additionally, its only some of the services each run on several different machines.
In my cloudflared container logs I cant see any errors. In traefik however I do see errors.
Error calling ``http://bouncer-traefik:8080/api/v1/forwardAuth`` error="Get \"``http://bouncer-traefik:8080/api/v1/forwardAuth\``": context canceled" middlewareName=crowdsec-bouncer@file middlewareType=ForwardAuth
My traefik.yml is as follows
global:
checkNewVersion: true
sendAnonymousUsage: false
api:
dashboard: true
debug: true
log:
level: DEBUG
filePath: "/var/log/traefik/traefik.log"
accessLog:
filePath: "/var/log/traefik/access.log"
entryPoints:
http:
address: ":80"
http:
# middlewares:
# - crowdsec-bouncer@file
redirections:
entryPoint:
to: https
scheme: https
https:
address: ":443"
http:
middlewares:
# - crowdsec-bouncer@file
- default-headers@file
- https-redirectscheme@file
mqtt:
address: ":1883"
mqtts:
address: ":8883"
certificatesResolvers:
cloudflare:
acme:
email: <email>
storage: /config/acme.json
dnsChallenge:
provider: cloudflare
resolvers:
- "<unbound>:53"
serversTransports:
insecure-selfsigned:
insecureSkipVerify: true
downgradeHTTP2:
forwardedHeaders:
trustedIPs:
- "0.0.0.0/0"
protocols:
- http/1.1
providers:
docker:
endpoint: "unix:///var/run/docker.sock"
exposedByDefault: false
file:
directory: /config
watch: true
http:
middlewares:
https-redirectscheme:
redirectScheme:
scheme: https
permanent: true
Initially I had the crowdsec bouncers in the entrypoints as well but I commented them out to try to add them manually to the routers down the line to see if the error is there.
My general config.yml looks like
http:
serversTransports:
insecure-selfsigned:
insecureSkipVerify: true
middlewares:
crowdsec-bouncer:
forwardauth:
address: http://bouncer-traefik:8080/api/v1/forwardAuth
trustForwardHeader: true
authRequestHeaders:
- Host
authResponseHeaders:
- X-Forwarded-User
- X-Forwarded-Groups
https-redirectscheme:
redirectScheme:
scheme: https
permanent: true
default-headers:
headers:
frameDeny: false
browserXssFilter: true
contentTypeNosniff: true
# contentSecurityPolicy: "frame-ancestors 'self'";
forceSTSHeader: true
stsIncludeSubdomains: true
stsPreload: true
stsSeconds: 15552000
customFrameOptionsValue: "ALLOW-FROM *"
customRequestHeaders:
X-Forwarded-Proto: https
secured:
chain:
middlewares:
- default-headers
routers:
<service>:
entryPoints:
- "https"
rule: "Host(`<service>.<domain>`)"
tls:
certResolver: cloudflare
service: <service>
services:
<service>:
loadBalancer:
servers:
- url: "http://<server_ip>[:<port>]"
passHostHeader: true
I have verified with a separate curl container connected to the proxy network that i can curl the bouncer. I made several modifications to authRequestHeaders and authResponseHeaders based on what I could find online. I am not sure where to look further.
When sparring with ai it pointed me in the direction of disabling http2, but that seemed to break more than it fixes (in typical AI fashion).
Thanks in advance!