HTTP Error 500 with CrowdSec middleware

Traefik Traefik v2
, ,
1

Hello. Yesterday i finally solved the issue with Traefik won't accept dynamic configuration file (with CrowdSec config in it). I solved it by adding tabs.

Anyway, after then as Traefik didn't throw error at me and just loaded configuration like shown below:

INFO[0000] Configuration loaded from file: /etc/traefik/traefik.yml

Then the webapp started to throw HTTP Error 500. So i ended up adding command to --log.level=DEBUG and hoping to see some error in traefik. Unfortunately, there is no error / debug log and instead just that one line with configuration loaded. Then i took down crowdsec containers to see if traefik would respond with error that it did not get response from crowdsec container. Still no error. Instead, when crowdsec is on, this is all the log i get from crowdsec:

Local agent already registered

Check if lapi needs to register an additional agent

sqlite database permissions updated

INFO[06-06-2023 13:21:02] hub index is up to date                      

INFO[06-06-2023 13:21:02] Wrote new 754885 bytes index to /etc/crowdsec/hub/.index.json 

Running: cscli  collections upgrade "crowdsecurity/linux"

INFO[06-06-2023 13:21:03] crowdsecurity/linux : up-to-date             

INFO[06-06-2023 13:21:03] Item 'crowdsecurity/linux' is up-to-date     

INFO[06-06-2023 13:21:03] Run 'sudo systemctl reload crowdsec' for the new configuration to be effective. 

Running: cscli  parsers upgrade "crowdsecurity/whitelists"

INFO[06-06-2023 13:21:03] crowdsecurity/whitelists : up-to-date        

INFO[06-06-2023 13:21:03] Item 'crowdsecurity/whitelists' is up-to-date 

INFO[06-06-2023 13:21:03] Run 'sudo systemctl reload crowdsec' for the new configuration to be effective. 

Running: cscli  parsers install "crowdsecurity/docker-logs"

WARN[06-06-2023 13:21:05] crowdsecurity/docker-logs : overwrite        

INFO[06-06-2023 13:21:05] Enabled crowdsecurity/docker-logs            

INFO[06-06-2023 13:21:05] Run 'sudo systemctl reload crowdsec' for the new configuration to be effective. 

Running: cscli  parsers install "crowdsecurity/cri-logs"

WARN[06-06-2023 13:21:06] crowdsecurity/cri-logs : overwrite           

INFO[06-06-2023 13:21:06] Enabled crowdsecurity/cri-logs               

INFO[06-06-2023 13:21:06] Run 'sudo systemctl reload crowdsec' for the new configuration to be effective. 

Running: cscli  collections install "crowdsecurity/linux"

WARN[06-06-2023 13:21:07] crowdsecurity/syslog-logs : overwrite        

WARN[06-06-2023 13:21:07] crowdsecurity/geoip-enrich : overwrite       

WARN[06-06-2023 13:21:07] crowdsecurity/dateparse-enrich : overwrite   

WARN[06-06-2023 13:21:07] crowdsecurity/sshd-logs : overwrite          

WARN[06-06-2023 13:21:07] crowdsecurity/ssh-bf : overwrite             

WARN[06-06-2023 13:21:07] crowdsecurity/ssh-slow-bf : overwrite        

WARN[06-06-2023 13:21:07] crowdsecurity/sshd : overwrite               

WARN[06-06-2023 13:21:07] crowdsecurity/sshd : overwrite               

WARN[06-06-2023 13:21:07] crowdsecurity/linux : overwrite              

INFO[06-06-2023 13:21:07] /etc/crowdsec/collections/sshd.yaml already exists. 

INFO[06-06-2023 13:21:07] /etc/crowdsec/collections/linux.yaml already exists. 

INFO[06-06-2023 13:21:07] Enabled crowdsecurity/linux                  

INFO[06-06-2023 13:21:07] Run 'sudo systemctl reload crowdsec' for the new configuration to be effective. 

Running: cscli  collections install "crowdsecurity/traefik"

WARN[06-06-2023 13:21:08] crowdsecurity/traefik-logs : overwrite       

WARN[06-06-2023 13:21:08] crowdsecurity/http-logs : overwrite          

WARN[06-06-2023 13:21:08] crowdsecurity/http-crawl-non_statics : overwrite 

WARN[06-06-2023 13:21:08] crowdsecurity/http-probing : overwrite       

WARN[06-06-2023 13:21:08] crowdsecurity/http-bad-user-agent : overwrite 

WARN[06-06-2023 13:21:08] crowdsecurity/http-path-traversal-probing : overwrite 

WARN[06-06-2023 13:21:08] crowdsecurity/http-sensitive-files : overwrite 

WARN[06-06-2023 13:21:08] crowdsecurity/http-sqli-probing : overwrite  

WARN[06-06-2023 13:21:08] crowdsecurity/http-xss-probing : overwrite   

WARN[06-06-2023 13:21:08] crowdsecurity/http-backdoors-attempts : overwrite 

WARN[06-06-2023 13:21:08] ltsich/http-w00tw00t : overwrite             

WARN[06-06-2023 13:21:08] crowdsecurity/http-generic-bf : overwrite    

WARN[06-06-2023 13:21:08] crowdsecurity/http-open-proxy : overwrite    

WARN[06-06-2023 13:21:08] crowdsecurity/http-cve-2021-41773 : overwrite 

WARN[06-06-2023 13:21:08] crowdsecurity/http-cve-2021-42013 : overwrite 

WARN[06-06-2023 13:21:08] crowdsecurity/grafana-cve-2021-43798 : overwrite 

WARN[06-06-2023 13:21:08] crowdsecurity/vmware-vcenter-vmsa-2021-0027 : overwrite 

WARN[06-06-2023 13:21:08] crowdsecurity/fortinet-cve-2018-13379 : overwrite 

WARN[06-06-2023 13:21:08] crowdsecurity/pulse-secure-sslvpn-cve-2019-11510 : overwrite 

WARN[06-06-2023 13:21:09] crowdsecurity/f5-big-ip-cve-2020-5902 : overwrite 

WARN[06-06-2023 13:21:09] crowdsecurity/thinkphp-cve-2018-20062 : overwrite 

WARN[06-06-2023 13:21:09] crowdsecurity/apache_log4j2_cve-2021-44228 : overwrite 

WARN[06-06-2023 13:21:09] crowdsecurity/jira_cve-2021-26086 : overwrite 

WARN[06-06-2023 13:21:09] crowdsecurity/spring4shell_cve-2022-22965 : overwrite 

WARN[06-06-2023 13:21:09] crowdsecurity/vmware-cve-2022-22954 : overwrite 

WARN[06-06-2023 13:21:09] crowdsecurity/CVE-2022-37042 : overwrite     

WARN[06-06-2023 13:21:09] crowdsecurity/CVE-2022-41082 : overwrite     

WARN[06-06-2023 13:21:09] crowdsecurity/CVE-2022-35914 : overwrite     

WARN[06-06-2023 13:21:09] crowdsecurity/CVE-2022-40684 : overwrite     

WARN[06-06-2023 13:21:09] crowdsecurity/CVE-2022-26134 : overwrite     

WARN[06-06-2023 13:21:09] crowdsecurity/CVE-2022-42889 : overwrite     

WARN[06-06-2023 13:21:09] crowdsecurity/CVE-2022-41697 : overwrite     

WARN[06-06-2023 13:21:09] crowdsecurity/CVE-2022-46169 : overwrite     

WARN[06-06-2023 13:21:09] crowdsecurity/CVE-2022-44877 : overwrite     

WARN[06-06-2023 13:21:09] crowdsecurity/CVE-2019-18935 : overwrite     

WARN[06-06-2023 13:21:09] crowdsecurity/http-cve : overwrite           

WARN[06-06-2023 13:21:09] crowdsecurity/http-cve : overwrite           

WARN[06-06-2023 13:21:09] crowdsecurity/base-http-scenarios : overwrite 

WARN[06-06-2023 13:21:09] crowdsecurity/base-http-scenarios : overwrite 

WARN[06-06-2023 13:21:09] crowdsecurity/traefik : overwrite            

INFO[06-06-2023 13:21:09] /etc/crowdsec/collections/http-cve.yaml already exists. 

INFO[06-06-2023 13:21:09] /etc/crowdsec/collections/base-http-scenarios.yaml already exists. 

INFO[06-06-2023 13:21:09] /etc/crowdsec/collections/traefik.yaml already exists. 

INFO[06-06-2023 13:21:09] Enabled crowdsecurity/traefik                

INFO[06-06-2023 13:21:09] Run 'sudo systemctl reload crowdsec' for the new configuration to be effective. 

Running: cscli  collections install "crowdsecurity/http-cve"

WARN[06-06-2023 13:21:10] crowdsecurity/http-cve-2021-41773 : overwrite 

WARN[06-06-2023 13:21:10] crowdsecurity/http-cve-2021-42013 : overwrite 

WARN[06-06-2023 13:21:10] crowdsecurity/grafana-cve-2021-43798 : overwrite 

WARN[06-06-2023 13:21:10] crowdsecurity/vmware-vcenter-vmsa-2021-0027 : overwrite 

WARN[06-06-2023 13:21:10] crowdsecurity/fortinet-cve-2018-13379 : overwrite 

WARN[06-06-2023 13:21:10] crowdsecurity/pulse-secure-sslvpn-cve-2019-11510 : overwrite 

WARN[06-06-2023 13:21:10] crowdsecurity/f5-big-ip-cve-2020-5902 : overwrite 

WARN[06-06-2023 13:21:10] crowdsecurity/thinkphp-cve-2018-20062 : overwrite 

WARN[06-06-2023 13:21:10] crowdsecurity/apache_log4j2_cve-2021-44228 : overwrite 

WARN[06-06-2023 13:21:10] crowdsecurity/jira_cve-2021-26086 : overwrite 

WARN[06-06-2023 13:21:10] crowdsecurity/spring4shell_cve-2022-22965 : overwrite 

WARN[06-06-2023 13:21:10] crowdsecurity/vmware-cve-2022-22954 : overwrite 

WARN[06-06-2023 13:21:10] crowdsecurity/CVE-2022-37042 : overwrite     

WARN[06-06-2023 13:21:10] crowdsecurity/CVE-2022-41082 : overwrite     

WARN[06-06-2023 13:21:10] crowdsecurity/CVE-2022-35914 : overwrite     

WARN[06-06-2023 13:21:10] crowdsecurity/CVE-2022-40684 : overwrite     

WARN[06-06-2023 13:21:10] crowdsecurity/CVE-2022-26134 : overwrite     

WARN[06-06-2023 13:21:10] crowdsecurity/CVE-2022-42889 : overwrite     

WARN[06-06-2023 13:21:10] crowdsecurity/CVE-2022-41697 : overwrite     

WARN[06-06-2023 13:21:10] crowdsecurity/CVE-2022-46169 : overwrite     

WARN[06-06-2023 13:21:10] crowdsecurity/CVE-2022-44877 : overwrite     

WARN[06-06-2023 13:21:10] crowdsecurity/CVE-2019-18935 : overwrite     

WARN[06-06-2023 13:21:10] crowdsecurity/http-cve : overwrite           

INFO[06-06-2023 13:21:10] /etc/crowdsec/collections/http-cve.yaml already exists. 

INFO[06-06-2023 13:21:10] Enabled crowdsecurity/http-cve               

INFO[06-06-2023 13:21:10] Run 'sudo systemctl reload crowdsec' for the new configuration to be effective. 

Running: cscli  collections install "crowdsecurity/whitelist-good-actors"

WARN[06-06-2023 13:21:11] crowdsecurity/seo-bots-whitelist : overwrite 

WARN[06-06-2023 13:21:11] crowdsecurity/cdn-whitelist : overwrite      

WARN[06-06-2023 13:21:11] crowdsecurity/rdns : overwrite               

WARN[06-06-2023 13:21:11] crowdsecurity/whitelist-good-actors : overwrite 

INFO[06-06-2023 13:21:11] /etc/crowdsec/collections/whitelist-good-actors.yaml already exists. 

INFO[06-06-2023 13:21:11] Enabled crowdsecurity/whitelist-good-actors  

INFO[06-06-2023 13:21:11] Run 'sudo systemctl reload crowdsec' for the new configuration to be effective. 

Running: cscli  collections install "crowdsecurity/sshd"

WARN[06-06-2023 13:21:12] crowdsecurity/sshd-logs : overwrite          

WARN[06-06-2023 13:21:12] crowdsecurity/ssh-bf : overwrite             

WARN[06-06-2023 13:21:12] crowdsecurity/ssh-slow-bf : overwrite        

WARN[06-06-2023 13:21:13] crowdsecurity/sshd : overwrite               

INFO[06-06-2023 13:21:13] /etc/crowdsec/collections/sshd.yaml already exists. 

INFO[06-06-2023 13:21:13] Enabled crowdsecurity/sshd                   

INFO[06-06-2023 13:21:13] Run 'sudo systemctl reload crowdsec' for the new configuration to be effective. 

INFO[06-06-2023 13:21:13] Enabled feature flags: <none>                

INFO[06-06-2023 13:21:13] Crowdsec v1.5.2-4fbc3402fba932c8bd34b671527dcf7909d264c0 

INFO[06-06-2023 13:21:13] Loading prometheus collectors                

INFO[06-06-2023 13:21:13] Loading CAPI manager                         

INFO[06-06-2023 13:21:14] CAPI manager configured successfully         

INFO[06-06-2023 13:21:14] CrowdSec Local API listening on 0.0.0.0:8080 

INFO[06-06-2023 13:21:14] Start push to CrowdSec Central API (interval: 15s once, then 10s) 

INFO[06-06-2023 13:21:14] Start send metrics to CrowdSec Central API (interval: 20m8s once, then 30m0s) 

INFO[06-06-2023 13:21:14] last CAPI pull is newer than 1h30, skip.     

INFO[06-06-2023 13:21:14] Start pull from CrowdSec Central API (interval: 1h56m45s once, then 2h0m0s) 

INFO[06-06-2023 13:21:14] Loading grok library /etc/crowdsec/patterns  

INFO[06-06-2023 13:21:14] capi metrics: metrics sent successfully      

INFO[06-06-2023 13:21:14] Loading enrich plugins                       

INFO[06-06-2023 13:21:14] Successfully registered enricher 'GeoIpCity' 

INFO[06-06-2023 13:21:14] Successfully registered enricher 'GeoIpASN'  

INFO[06-06-2023 13:21:14] Successfully registered enricher 'IpToRange' 

INFO[06-06-2023 13:21:14] Successfully registered enricher 'reverse_dns' 

INFO[06-06-2023 13:21:14] Successfully registered enricher 'ParseDate' 

INFO[06-06-2023 13:21:14] Successfully registered enricher 'UnmarshalJSON' 

INFO[06-06-2023 13:21:14] Loading parsers from 9 files                 

INFO[06-06-2023 13:21:14] Loaded 1 parser nodes                         file=/etc/crowdsec/parsers/s00-raw/cri-logs.yaml stage=s00-raw

INFO[06-06-2023 13:21:14] Loaded 1 parser nodes                         file=/etc/crowdsec/parsers/s00-raw/docker-logs.yaml stage=s00-raw

INFO[06-06-2023 13:21:14] Loaded 2 parser nodes                         file=/etc/crowdsec/parsers/s00-raw/syslog-logs.yaml stage=s00-raw

INFO[06-06-2023 13:21:14] Loaded 1 parser nodes                         file=/etc/crowdsec/parsers/s01-parse/sshd-logs.yaml stage=s01-parse

INFO[06-06-2023 13:21:14] Loaded 1 parser nodes                         file=/etc/crowdsec/parsers/s01-parse/traefik-logs.yaml stage=s01-parse

INFO[06-06-2023 13:21:14] Loaded 1 parser nodes                         file=/etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml stage=s02-enrich

INFO[06-06-2023 13:21:14] Loaded 1 parser nodes                         file=/etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml stage=s02-enrich

INFO[06-06-2023 13:21:14] Loaded 1 parser nodes                         file=/etc/crowdsec/parsers/s02-enrich/http-logs.yaml stage=s02-enrich

INFO[06-06-2023 13:21:14] Loaded 1 parser nodes                         file=/etc/crowdsec/parsers/s02-enrich/whitelists.yaml stage=s02-enrich

INFO[06-06-2023 13:21:14] Loaded 10 nodes from 3 stages                

INFO[06-06-2023 13:21:14] Loading postoverflow parsers                 

INFO[06-06-2023 13:21:14] Loaded 1 parser nodes                         file=/etc/crowdsec/postoverflows/s00-enrich/rdns.yaml stage=s00-enrich

INFO[06-06-2023 13:21:14] Loaded 1 parser nodes                         file=/etc/crowdsec/postoverflows/s01-whitelist/cdn-whitelist.yaml stage=s01-whitelist

INFO[06-06-2023 13:21:14] Loaded 1 parser nodes                         file=/etc/crowdsec/postoverflows/s01-whitelist/seo-bots-whitelist.yaml stage=s01-whitelist

INFO[06-06-2023 13:21:14] Loaded 3 nodes from 2 stages                 

INFO[06-06-2023 13:21:14] Loading 35 scenario files                    

INFO[06-06-2023 13:21:14] Adding trigger bucket                         cfg=broken-silence file=/etc/crowdsec/scenarios/fortinet-cve-2018-13379.yaml name=crowdsecurity/fortinet-cve-2018-13379

INFO[06-06-2023 13:21:14] Adding leaky bucket                           cfg=autumn-morning file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf

INFO[06-06-2023 13:21:14] Adding leaky bucket                           cfg=ancient-silence file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf_user-enum

INFO[06-06-2023 13:21:14] Adding trigger bucket                         cfg=throbbing-feather file=/etc/crowdsec/scenarios/pulse-secure-sslvpn-cve-2019-11510.yaml name=crowdsecurity/pulse-secure-sslvpn-cve-2019-11510

INFO[06-06-2023 13:21:14] Adding trigger bucket                         cfg=lingering-sky file=/etc/crowdsec/scenarios/CVE-2022-26134.yaml name=crowdsecurity/CVE-2022-26134

INFO[06-06-2023 13:21:14] Adding leaky bucket                           cfg=lively-silence file=/etc/crowdsec/scenarios/http-path-traversal-probing.yaml name=crowdsecurity/http-path-traversal-probing

INFO[06-06-2023 13:21:14] Adding trigger bucket                         cfg=fragrant-leaf file=/etc/crowdsec/scenarios/jira_cve-2021-26086.yaml name=crowdsecurity/jira_cve-2021-26086

INFO[06-06-2023 13:21:14] Adding trigger bucket                         cfg=empty-fog file=/etc/crowdsec/scenarios/thinkphp-cve-2018-20062.yaml name=crowdsecurity/thinkphp-cve-2018-20062

INFO[06-06-2023 13:21:14] Adding leaky bucket                           cfg=billowing-frost file=/etc/crowdsec/scenarios/CVE-2022-41697.yaml name=crowdsecurity/CVE-2022-41697

INFO[06-06-2023 13:21:14] Adding trigger bucket                         cfg=fragrant-lake file=/etc/crowdsec/scenarios/CVE-2022-37042.yaml name=crowdsecurity/CVE-2022-37042

INFO[06-06-2023 13:21:14] Adding trigger bucket                         cfg=still-moon file=/etc/crowdsec/scenarios/vmware-cve-2022-22954.yaml name=crowdsecurity/vmware-cve-2022-22954

INFO[06-06-2023 13:21:14] Adding trigger bucket                         cfg=spring-thunder file=/etc/crowdsec/scenarios/f5-big-ip-cve-2020-5902.yaml name=crowdsecurity/f5-big-ip-cve-2020-5902

INFO[06-06-2023 13:21:14] Adding leaky bucket                           cfg=patient-bird file=/etc/crowdsec/scenarios/http-sqli-probing.yaml name=crowdsecurity/http-sqli-probbing-detection

INFO[06-06-2023 13:21:14] Adding trigger bucket                         cfg=restless-snowflake file=/etc/crowdsec/scenarios/apache_log4j2_cve-2021-44228.yaml name=crowdsecurity/apache_log4j2_cve-2021-44228

INFO[06-06-2023 13:21:14] Adding trigger bucket                         cfg=weathered-star file=/etc/crowdsec/scenarios/http-cve-2021-42013.yaml name=crowdsecurity/http-cve-2021-42013

INFO[06-06-2023 13:21:14] Adding leaky bucket                           cfg=falling-meadow file=/etc/crowdsec/scenarios/http-sensitive-files.yaml name=crowdsecurity/http-sensitive-files

INFO[06-06-2023 13:21:14] Adding trigger bucket                         cfg=summer-river file=/etc/crowdsec/scenarios/CVE-2022-40684.yaml name=crowdsecurity/fortinet-cve-2022-40684

INFO[06-06-2023 13:21:14] Adding leaky bucket                           cfg=solitary-tree file=/etc/crowdsec/scenarios/http-crawl-non_statics.yaml name=crowdsecurity/http-crawl-non_statics

INFO[06-06-2023 13:21:14] Adding trigger bucket                         cfg=wispy-butterfly file=/etc/crowdsec/scenarios/http-cve-2021-41773.yaml name=crowdsecurity/http-cve-2021-41773

INFO[06-06-2023 13:21:14] Adding trigger bucket                         cfg=wandering-water file=/etc/crowdsec/scenarios/spring4shell_cve-2022-22965.yaml name=crowdsecurity/spring4shell_cve-2022-22965

INFO[06-06-2023 13:21:14] Adding leaky bucket                           cfg=polished-snow file=/etc/crowdsec/scenarios/CVE-2022-46169.yaml name=crowdsecurity/CVE-2022-46169-bf

INFO[06-06-2023 13:21:14] Adding trigger bucket                         cfg=empty-pond file=/etc/crowdsec/scenarios/CVE-2022-46169.yaml name=crowdsecurity/CVE-2022-46169-cmd

INFO[06-06-2023 13:21:14] Adding leaky bucket                           cfg=quiet-water file=/etc/crowdsec/scenarios/ssh-slow-bf.yaml name=crowdsecurity/ssh-slow-bf

INFO[06-06-2023 13:21:14] Adding leaky bucket                           cfg=blue-water file=/etc/crowdsec/scenarios/ssh-slow-bf.yaml name=crowdsecurity/ssh-slow-bf_user-enum

INFO[06-06-2023 13:21:14] Adding trigger bucket                         cfg=bold-forest file=/etc/crowdsec/scenarios/CVE-2019-18935.yaml name=crowdsecurity/CVE-2019-18935

INFO[06-06-2023 13:21:14] Adding leaky bucket                           cfg=wandering-water file=/etc/crowdsec/scenarios/http-generic-bf.yaml name=crowdsecurity/http-generic-bf

INFO[06-06-2023 13:21:14] Adding leaky bucket                           cfg=billowing-grass file=/etc/crowdsec/scenarios/http-generic-bf.yaml name=LePresidente/http-generic-401-bf

INFO[06-06-2023 13:21:14] Adding leaky bucket                           cfg=billowing-paper file=/etc/crowdsec/scenarios/http-generic-bf.yaml name=LePresidente/http-generic-403-bf

INFO[06-06-2023 13:21:14] Adding leaky bucket                           cfg=sparkling-wildflower file=/etc/crowdsec/scenarios/http-probing.yaml name=crowdsecurity/http-probing

INFO[06-06-2023 13:21:14] Adding leaky bucket                           cfg=empty-sea file=/etc/crowdsec/scenarios/http-xss-probing.yaml name=crowdsecurity/http-xss-probbing

INFO[06-06-2023 13:21:14] Adding trigger bucket                         cfg=old-sun file=/etc/crowdsec/scenarios/CVE-2022-35914.yaml name=crowdsecurity/CVE-2022-35914

INFO[06-06-2023 13:21:14] Adding trigger bucket                         cfg=green-frog file=/etc/crowdsec/scenarios/CVE-2022-42889.yaml name=crowdsecurity/CVE-2022-42889

INFO[06-06-2023 13:21:14] Adding trigger bucket                         cfg=misty-fire file=/etc/crowdsec/scenarios/grafana-cve-2021-43798.yaml name=crowdsecurity/grafana-cve-2021-43798

INFO[06-06-2023 13:21:14] Adding leaky bucket                           cfg=frosty-haze file=/etc/crowdsec/scenarios/http-backdoors-attempts.yaml name=crowdsecurity/http-backdoors-attempts

INFO[06-06-2023 13:21:14] Adding trigger bucket                         cfg=still-frost file=/etc/crowdsec/scenarios/CVE-2022-44877.yaml name=crowdsecurity/CVE-2022-44877

INFO[06-06-2023 13:21:14] Adding trigger bucket                         cfg=shy-paper file=/etc/crowdsec/scenarios/http-open-proxy.yaml name=crowdsecurity/http-open-proxy

INFO[06-06-2023 13:21:14] Adding trigger bucket                         cfg=wandering-sun file=/etc/crowdsec/scenarios/CVE-2022-41082.yaml name=crowdsecurity/CVE-2022-41082

INFO[06-06-2023 13:21:14] Adding leaky bucket                           cfg=broken-wind file=/etc/crowdsec/scenarios/http-bad-user-agent.yaml name=crowdsecurity/http-bad-user-agent

INFO[06-06-2023 13:21:14] Adding trigger bucket                         cfg=summer-haze file=/etc/crowdsec/scenarios/http-w00tw00t.yaml name=ltsich/http-w00tw00t

INFO[06-06-2023 13:21:14] Adding trigger bucket                         cfg=delicate-mountain file=/etc/crowdsec/scenarios/vmware-vcenter-vmsa-2021-0027.yaml name=crowdsecurity/vmware-vcenter-vmsa-2021-0027

WARN[06-06-2023 13:21:14] Loaded 40 scenarios                          

INFO[06-06-2023 13:21:14] loading acquisition file : /etc/crowdsec/acquis.yaml 

WARN[06-06-2023 13:21:14] No matching files for pattern /var/log/nginx/*.log  type=file

WARN[06-06-2023 13:21:14] No matching files for pattern ./tests/nginx/nginx.log  type=file

INFO[06-06-2023 13:21:14] Adding file /var/log/auth.log to datasources  type=file

WARN[06-06-2023 13:21:14] No matching files for pattern /var/log/syslog  type=file

WARN[06-06-2023 13:21:14] No matching files for pattern /var/log/apache2/*.log  type=file

INFO[06-06-2023 13:21:14] Starting processing data                     

INFO[06-06-2023 13:21:14] 127.0.0.1 - [Tue, 06 Jun 2023 13:21:14 CEST] "POST /v1/watchers/login HTTP/1.1 200 89.216413ms "crowdsec/v1.5.2-4fbc3402fba932c8bd34b671527dcf7909d264c0" " 

INFO[06-06-2023 13:22:14] 127.0.0.1 - [Tue, 06 Jun 2023 13:22:14 CEST] "GET /v1/heartbeat HTTP/1.1 200 21.667476ms "crowdsec/v1.5.2-4fbc3402fba932c8bd34b671527dcf7909d264c0" " 

INFO[06-06-2023 13:23:14] 127.0.0.1 - [Tue, 06 Jun 2023 13:23:14 CEST] "GET /v1/heartbeat HTTP/1.1 200 11.56223ms "crowdsec/v1.5.2-4fbc3402fba932c8bd34b671527dcf7909d264c0" " 

INFO[06-06-2023 13:24:14] 127.0.0.1 - [Tue, 06 Jun 2023 13:24:14 CEST] "GET /v1/heartbeat HTTP/1.1 200 11.988899ms "crowdsec/v1.5.2-4fbc3402fba932c8bd34b671527dcf7909d264c0" " 

INFO[06-06-2023 13:25:14] 127.0.0.1 - [Tue, 06 Jun 2023 13:25:14 CEST] "GET /v1/heartbeat HTTP/1.1 200 19.92613ms "crowdsec/v1.5.2-4fbc3402fba932c8bd34b671527dcf7909d264c0" " 

INFO[06-06-2023 13:26:14] 127.0.0.1 - [Tue, 06 Jun 2023 13:26:14 CEST] "GET /v1/heartbeat HTTP/1.1 200 12.96186ms "crowdsec/v1.5.2-4fbc3402fba932c8bd34b671527dcf7909d264c0" " 

INFO[06-06-2023 13:27:14] 127.0.0.1 - [Tue, 06 Jun 2023 13:27:14 CEST] "GET /v1/heartbeat HTTP/1.1 200 11.802771ms "crowdsec/v1.5.2-4fbc3402fba932c8bd34b671527dcf7909d264c0" " 

INFO[06-06-2023 13:28:14] 127.0.0.1 - [Tue, 06 Jun 2023 13:28:14 CEST] "GET /v1/heartbeat HTTP/1.1 200 16.429472ms "crowdsec/v1.5.2-4fbc3402fba932c8bd34b671527dcf7909d264c0" " 

INFO[06-06-2023 13:29:14] 127.0.0.1 - [Tue, 06 Jun 2023 13:29:14 CEST] "GET /v1/heartbeat HTTP/1.1 200 13.383891ms "crowdsec/v1.5.2-4fbc3402fba932c8bd34b671527dcf7909d264c0" " 

INFO[06-06-2023 13:30:14] 127.0.0.1 - [Tue, 06 Jun 2023 13:30:14 CEST] "GET /v1/heartbeat HTTP/1.1 200 29.979863ms "crowdsec/v1.5.2-4fbc3402fba932c8bd34b671527dcf7909d264c0" " 

INFO[06-06-2023 13:31:14] 127.0.0.1 - [Tue, 06 Jun 2023 13:31:14 CEST] "GET /v1/heartbeat HTTP/1.1 200 11.057494ms "crowdsec/v1.5.2-4fbc3402fba932c8bd34b671527dcf7909d264c0" "

Is there something special that i should consider?

Traefik docker compose file:

version: "3"
services:

  traefik:
    image: "traefik:latest"
    container_name: "traefik"
    command:
      - "--log.level=DEBUG"
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    stdin_open: true # docker run -i
    tty: true        # docker run -t
    ports:
      - "80:80"
      - "443:443"
      - "4130:8080"
    environment:
      - CF_API_EMAIL=redacted
      - CF_API_KEY=redacted
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - /home/docker/docker_traefik:/etc/traefik
      - traefik_ssl_certs:/etc/traefik/certs
      
volumes:
  traefik_ssl_certs:

And crowdsec docker compose file:

version: "3.3"
services:
  crowdsec:
    container_name: crowdsec
    image: crowdsecurity/crowdsec:latest
    stdin_open: true # docker run -i
    tty: true        # docker run -t
    environment:
      GID: "1001"
      COLLECTIONS: "crowdsecurity/linux crowdsecurity/traefik crowdsecurity/http-cve crowdsecurity/whitelist-good-actors crowdsecurity/sshd"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /home/docker/docker_crowdsec/crowdsec:/etc/crowdsec
      - /var/log/auth.log:/var/log/auth.log:ro
      - /var/log/crowdsec:/var/log/crowdsec:ro
      - /home/docker/docker_crowdsec/database:/var/lib/crowdsec/data
    restart: unless-stopped
    security_opt:
      - no-new-privileges=true
    hostname: crowdsec
    
  bouncer-traefik:
    container_name: crowdsec-bouncer-traefik
    image: fbonalair/traefik-crowdsec-bouncer:latest
    stdin_open: true # docker run -i
    tty: true        # docker run -t
    environment:
      CROWDSEC_BOUNCER_API_KEY: redacted # the api key needs to be created of the crowdsec container with `docker compose exec -t crowdsec cscli bouncers add bouncer-traefik`
      CROWDSEC_AGENT_HOST: crowdsec:8080  
    restart: unless-stopped
    depends_on:
      - crowdsec
    hostname: crowdsec-bouncer-traefik

PS: Just a thought, couldn't it be caused by not having traefik and crowdsec in the same docker network?

2

Please share your Traefik static and dynamic config.

Logging does not work because you probably have traefik.yml and Traefik only accepts a single static config.

So you need to enable logging in traefik.yml.

3

I apologise. I knew i forgot someting.

Static:

global:
  checkNewVersion: true
  sendAnonymousUsage: false  # true by default

# (Optional) Log information
# ---
# log:
#  level: ERROR  # DEBUG, INFO, WARNING, ERROR, CRITICAL
#   format: common  # common, json, logfmt
#   filePath: /var/log/traefik/traefik.log

# (Optional) Accesslog
# ---
# accesslog:
  # format: common  # common, json, logfmt
  # filePath: /var/log/traefik/access.log

# (Optional) Enable API and Dashboard
# ---
api:
 dashboard: true  # true by default
 insecure: true  # Don't do this in production!

# Entry Points configuration
# ---
entryPoints:
  web:
    address: :80
    # (Optional) Redirect to HTTPS
    # ---
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https

  websecure:
    address: :443
    http:
      middlewares:
        - crowdsec-bouncer@file

# Insecure Skip Verify
# ---
serversTransport:
  insecureSkipVerify: true

# Configure your CertificateResolver here...
# ---
certificatesResolvers:
# Cloudflare (local domain)
  cloudflare:
    acme:
      email: redacted
      storage: /etc/traefik/certs/acme.json
      dnsChallenge:
        provider: cloudflare
        resolvers:
          - "1.1.1.1:53"
          - "1.0.0.1:53"

# LetsEncrypt (public domain)
  staging:
    acme:
      email: redacted
      storage: /etc/traefik/certs/acme.json
      caServer: "https://acme-staging-v02.api.letsencrypt.org/directory"
      httpChallenge:
        entryPoint: web

  production:
    acme:
      email: redacted
      storage: /etc/traefik/certs/acme.json
      caServer: "https://acme-v02.api.letsencrypt.org/directory"
      httpChallenge:
        entryPoint: web

# (Optional) Overwrite Default Certificates
# tls:
#   stores:
#     default:
#       defaultCertificate:
#         certFile: /etc/traefik/certs/cert.pem
#         keyFile: /etc/traefik/certs/cert-key.pem
# (Optional) Disable TLS version 1.0 and 1.1
#   options:
#     default:
#       minVersion: VersionTLS12

providers:
  docker:
    exposedByDefault: false  # Default is true
  file:
    # watch for dynamic configuration changes
    filename: "/etc/traefik/dynamic/dynamic-config.yml"
    watch: true

and dynamic: (located in ./dynamic/dynamic-config.yml)

# Middlewares
# ---
http:
  middlewares:
    crowdsec-bouncer:
      forwardauth:
        address: http://crowdsec-bouncer-traefik:8080/api/v1/forwardAuth
        trustForwardHeader: true
4

I finally got into this. This is the log (some lines redacted due to privacy).

It seems like it can't get the certificate:

time="2023-06-07T11:37:27Z" level=debug msg="Looking for provided certificate(s) to validate [\"local.example.eu\" \"*.local.example.eu\"]..." providerName=cloudflare.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory"
time="2023-06-07T11:37:27Z" level=debug msg="No ACME certificate generation required for domains [\"local.example.eu\" \"*.local.example.eu\"]." ACME CA="https://acme-v02.api.letsencrypt.org/directory" providerName=cloudflare.acme
time="2023-06-07T11:37:27Z" level=debug msg="Looking for provided certificate(s) to validate [\"local.example.eu\" \"*.local.example.eu\"]..." ACME CA="https://acme-v02.api.letsencrypt.org/directory" providerName=cloudflare.acme
time="2023-06-07T11:37:27Z" level=debug msg="Trying to challenge certificate for domain [prowlarr.services.example.eu] found in HostSNI rule" providerName=staging.acme routerName=prowlarr@docker rule="Host(`prowlarr.services.example.eu`)" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory"
time="2023-06-07T11:37:27Z" level=debug msg="No ACME certificate generation required for domains [\"local.example.eu\" \"*.local.example.eu\"]." ACME CA="https://acme-v02.api.letsencrypt.org/directory" providerName=cloudflare.acme
time="2023-06-07T11:37:27Z" level=debug msg="Trying to challenge certificate for domain [prowlarr.services.example.eu] found in HostSNI rule" rule="Host(`prowlarr.services.example.eu`)" routerName=websecure-prowlarr@docker ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=staging.acme
time="2023-06-07T11:37:27Z" level=debug msg="Looking for provided certificate(s) to validate [\"prowlarr.services.example.eu\"]..." ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=staging.acme rule="Host(`prowlarr.services.example.eu`)" routerName=websecure-prowlarr@docker
time="2023-06-07T11:37:27Z" level=debug msg="No ACME certificate generation required for domains [\"prowlarr.services.example.eu\"]." ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=staging.acme rule="Host(`prowlarr.services.example.eu`)" routerName=websecure-prowlarr@docker
time="2023-06-07T11:37:27Z" level=debug msg="Looking for provided certificate(s) to validate [\"prowlarr.services.example.eu\"]..." routerName=prowlarr@docker rule="Host(`prowlarr.services.example.eu`)" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=staging.acme
time="2023-06-07T11:37:27Z" level=debug msg="No ACME certificate generation required for domains [\"prowlarr.services.example.eu\"]." routerName=prowlarr@docker rule="Host(`prowlarr.services.example.eu`)" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=staging.acme

And also there's error while calling local DNS:

time="2023-06-07T11:37:29Z" level=debug msg="Error calling http://crowdsec-bouncer-traefik:8080/api/v1/forwardAuth. Cause: Get \"http://crowdsec-bouncer-traefik:8080/api/v1/forwardAuth\": dial tcp: lookup crowdsec-bouncer-traefik on 127.0.0.11:53: no such host" middlewareName=crowdsec-bouncer@file middlewareType=ForwardedAuthType
time="2023-06-07T11:37:30Z" level=debug msg="Error calling http://crowdsec-bouncer-traefik:8080/api/v1/forwardAuth. Cause: Get \"http://crowdsec-bouncer-traefik:8080/api/v1/forwardAuth\": dial tcp: lookup crowdsec-bouncer-traefik on 127.0.0.11:53: no such host" middlewareName=crowdsec-bouncer@file middlewareType=ForwardedAuthType
time="2023-06-07T11:37:31Z" level=debug msg="Error calling http://crowdsec-bouncer-traefik:8080/api/v1/forwardAuth. Cause: Get \"http://crowdsec-bouncer-traefik:8080/api/v1/forwardAuth\": dial tcp: lookup crowdsec-bouncer-traefik on 127.0.0.11:53: no such host" middlewareType=ForwardedAuthType middlewareName=crowdsec-bouncer@file

Resulting in unknown certificate while serving it:

time="2023-06-07T11:37:37Z" level=debug msg="http: TLS handshake error from 192.168.1.1:58774: remote error: tls: unknown certificate"
time="2023-06-07T11:37:37Z" level=debug msg="http: TLS handshake error from 192.168.1.1:58773: remote error: tls: unknown certificate"

Any tips how could i fix it?

PS: This setup works fine if i undo the crowdsec part. So certificates shouldn't be an issue maybe?

5

@bluepuma77 are you aware of any kind of solution?

6

Fix one at a time.

does not exist in Traefik internal DNS because the service in docker-compose.yml is only called crowdsec

7

Thank you so much for reply. I finally solved the issue with HTTP Error 500 by combining those two docker-compose files. I'm not sure how docker compose works in networking part but as i combined that two docker-compose files, i was able to connect to those webapps with SSL certs.

8

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.