Docker stack(wazuh) with several ports

Traefik Traefik v2
1

Hii
We are working on a docker-compose config containing 3 services.
Those services must be accessed via several ports.
Service 1:

  • 514 UDP
  • 1514 TCP
  • 1515 TCP
  • 55000 TCP

Service 2:

  • 9200 TCP

Service 3:

  • 5601

We did setup Service/container 3, as usual no issues.
But the way to setup the other ports is a mystery.
We tried many-many versions but none of which worked.
Setup:
Proxmox server running 2 LXC:

  • Traefik
  • Wazuh

Please help us to figure out the right way to label the services/port rules.
I cant add my 2 config files atm because it gets flagged as spam...

Ty for your time and kind help

2

Are those ports using plain TCP or HTTP? Is TLS used externally and internally to forward? Where are certs coming from?

3

Hi
Ty for you answer let me see if i can answer those correctly.

  • Ports are probably plain TCP i think. Locally those worked.
  • TLS is acquired by dns challenge from dynu ddns service and stored localy in the letscencrypt json file.
  • The certificates are forwarded to services by these labels:
      - traefik.http.routers.wazuhdash.tls.certresolver=letsencrypt
      - traefik.http.routers.wazuhdash.tls.domains[0].main=wazuh.xxx.webredirect.org
      - traefik.http.routers.wazuhdash.tls.domains[0].sans=*.wazuh.xxx.webredirect.org

  websecure:
    address: ":443"
    http:
      middlewares:
        - crowdsec-bouncer@file
      tls:
        certResolver: letsencrypt
        domains:
          - main: xxx.webredirect.org
            sans:
              - '*.xxx.webredirect.org'

certificatesResolvers:
  letsencrypt:
    acme:
      email: xxx@protonmail.com
      storage: /letsencrypt/acme.json
      dnsChallenge:
        provider: dynu
        resolvers:
          - "1.1.1.1:53"
          - "1.0.0.1:53"
4

If you want the ports to be available externally, then you need to open the ports in Traefik docker-compose.yml and create matching entrypoints in Traefik static config.

Then you create the dynamic config with labels, make sure to use the right protocol in router (doc) and add the target port in service in case the image exposes multiple.

Check and compare to simple Traefik example and simple Traefik TCP example to see full working configurations.

For further questions, please share your full Traefik static and dynamic config, and docker-compose.yml.

5

Ty for the direction
So all in all

  • Forward the ports in the router
  • Add these ports as entrypoints in traefik static config (similar as 80,443)
  • Add the dynamic configuration ONLY at the docker-compose file for the services by using labels.

TY TY !!

6

Hi i did my best to follow the examples and documentation, but i get a little bit lost.

  • Added/Exposed the required ports in the Traefik docker-compose.yml
ports:
      - "80:80"
      - "443:443"
      - "6688:8080"
      - "514:514"
      - "1514:1514"
      - "1515:1515"
      - "9200:9200"
      - "55000:55000"
  • Set the entry points in traefik static configuration file(traefik.yml)
api:
  insecure: true

providers:
  docker:
    exposedbydefault: false
    network: proxy
  file:
    filename: /etc/traefik/config.yml
    watch: true

entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entrypoint:
          to: websecure
          scheme: https
      middlewares:
        - crowdsec-bouncer@file
  websecure:
    address: ":443"
    http:
      middlewares:
        - crowdsec-bouncer@file
      tls:
        certResolver: letsencrypt
        domains:
          - main: xxx.webredirect.org
            sans:
              - '*.xxx.webredirect.org'
#--Wazuh entry point ports--
  syslog:
    address: ":514/udp"
  logstash:
    address: ":1514/tcp"
  logstash-beats:
    address: ":1515/tcp"
  elasticsearch:
    address: ":9200/tcp"
  kibana:
    address: ":55000/tcp"
#---------------
ping: true

certificatesResolvers:
  letsencrypt:
    acme:
      email: xxx@protonmail.com
      storage: /letsencrypt/acme.json
      dnsChallenge:
        provider: dynu
        resolvers:
          - "1.1.1.1:53"
          - "1.0.0.1:53"

log:
  level: "INFO"
  filePath: "/var/log/traefik/traefik.log"
accessLog:
  filePath: "/var/log/traefik/access.log"
  • Added the routers in the dynamic configuration file(config.yml)
http:
  middlewares:
    crowdsec-bouncer:
      forwardauth:
        address: http://bouncer-traefik:8080/api/v1/forwardAuth
        trustForwardHeader: true

  routers:
#---Wazuh dashboard on port 5601
    wazuh-manager:
      rule: "Host(`wazuh.xxx.webredirect.org`)"
      service: wazuhmanager
#---Wazuh plain UDP/TCP ports:
    Router-514:
      entryPoints:
        - "websecure"
        - "syslog"
      rule: "Host(`wazuh.xxx.webredirect.org`)"
      service: "wz1514"
    Router-1514:
      entryPoints:
        - "websecure"
        - "logstash"
      rule: "Host(`wazuh.xxx.webredirect.org`)"
      service: "wz1514"
    Router-1515:
      entryPoints:
        - "websecure"
        - "logstash-beats"
      rule: "Host(`wazuh.xxx.webredirect.org`)"
      service: "wz1515"
    Router-9200:
      entryPoints:
        - "websecure"
        - "elasticsearch"
      rule: "Host(`wazuh.xxx.webredirect.org`)"
      service: "wz9200"
    Router-55000:
      entryPoints:
        - "websecure"
        - "kibana"
      rule: "Host(`wazuh.xxx.webredirect.org`)"
      service: "wz55000"
#---Services---
  services:
#---Wazuh dash
    wazuhmanager:
      loadbalancer:
        serversTransport: wazuhtransport
        servers:
          - url: "https://192.168.0.219:5601"

  serversTransports:
    wazuhtransport:
      insecureSkipVerify: true

And my docker-compose file with the services is a dizaster because im confused about the "services" declaraton.
Therefore i cant even imagine how the actual labels in the wazuh docker-compose will look like.

7

So at the moment im "running" something like this and for sure you will see what epic mess i made with the labels for the first container.

version: '3.7'

services:
  wazuh.manager:
    image: wazuh/wazuh-manager:4.7.1
    hostname: wazuh.manager
    restart: always
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 655360
        hard: 655360
    networks:
      - proxy
    ports:
      - "1514:1514"
      - "1515:1515"
      - "514:514/udp"
      - "55000:55000"
    environment:
      - INDEXER_URL=https://wazuh.indexer:9200
      - INDEXER_USERNAME=admin
      - INDEXER_PASSWORD=uxxxDb%!
      - FILEBEAT_SSL_VERIFICATION_MODE=full
      - SSL_CERTIFICATE_AUTHORITIES=/etc/ssl/root-ca.pem
      - SSL_CERTIFICATE=/etc/ssl/filebeat.pem
      - SSL_KEY=/etc/ssl/filebeat.key
      - API_USERNAME=wazuh-wui
      - API_PASSWORD=MyS3cr37P450r.*-
    volumes:
      - wazuh_api_configuration:/var/ossec/api/configuration
      - wazuh_etc:/var/ossec/etc
      - wazuh_logs:/var/ossec/logs
      - wazuh_queue:/var/ossec/queue
      - wazuh_var_multigroups:/var/ossec/var/multigroups
      - wazuh_integrations:/var/ossec/integrations
      - wazuh_active_response:/var/ossec/active-response/bin
      - wazuh_agentless:/var/ossec/agentless
      - wazuh_wodles:/var/ossec/wodles
      - filebeat_etc:/etc/filebeat
      - filebeat_var:/var/lib/filebeat
      - ./config/wazuh_indexer_ssl_certs/root-ca-manager.pem:/etc/ssl/root-ca.pem
      - ./config/wazuh_indexer_ssl_certs/wazuh.manager.pem:/etc/ssl/filebeat.pem
      - ./config/wazuh_indexer_ssl_certs/wazuh.manager-key.pem:/etc/ssl/filebeat.key
      - ./config/wazuh_cluster/wazuh_manager.conf:/wazuh-config-mount/etc/ossec.conf
    labels:
      - traefik.enable=true
      - traefik.http.routers.wz514.rule=Host(`wazuh.xxx.webredirect.org`)
      - traefik.http.routers.wz514.entrypoints=ep1
      - traefik.http.routers.wz514.service=svc_wz514
      - traefik.http.services.wz514.loadbalancer.server.port=514
      - traefik.http.routers.wz1514.rule=Host(`wazuh.xxx.webredirect.org`)
      - traefik.http.routers.wz1514.entrypoints=ep2
      - traefik.http.routers.wz1514.service=svc_wz1514
      - traefik.http.services.wz1514.loadbalancer.server.port=1514
      - traefik.http.routers.wz1515.rule=Host(`wazuh.xxx.webredirect.org`)
      - traefik.http.routers.wz1515.entrypoints=ep3
      - traefik.http.routers.wz1515.service=svc_wz1515
      - traefik.http.services.wz1515.loadbalancer.server.port=1515
      - traefik.http.routers.wz55000.rule=Host(`wazuh.xxx.webredirect.org`)
      - traefik.http.routers.wz55000.entrypoints=ep4
      - traefik.http.routers.wz55000.service=svc_wz55000
      - traefik.http.services.wz55000.loadbalancer.server.port=55000

  wazuh.indexer:
    image: wazuh/wazuh-indexer:4.7.1
    hostname: wazuh.indexer
    restart: always
    networks:
      - proxy
    ports:
      - "9200:9200"
    environment:
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - wazuh-indexer-data:/var/lib/wazuh-indexer
      - ./config/wazuh_indexer_ssl_certs/root-ca.pem:/usr/share/wazuh-indexer/certs/root-ca.pem
      - ./config/wazuh_indexer_ssl_certs/wazuh.indexer-key.pem:/usr/share/wazuh-indexer/certs/wazuh.indexer.key
      - ./config/wazuh_indexer_ssl_certs/wazuh.indexer.pem:/usr/share/wazuh-indexer/certs/wazuh.indexer.pem
      - ./config/wazuh_indexer_ssl_certs/admin.pem:/usr/share/wazuh-indexer/certs/admin.pem
      - ./config/wazuh_indexer_ssl_certs/admin-key.pem:/usr/share/wazuh-indexer/certs/admin-key.pem
      - ./config/wazuh_indexer/wazuh.indexer.yml:/usr/share/wazuh-indexer/opensearch.yml
      - ./config/wazuh_indexer/internal_users.yml:/usr/share/wazuh-indexer/opensearch-security/internal_users.yml
    labels:
      - traefik.enable=true
      - traefik.http.routers.wazuhindexer.rule=Host(`wazuh.xxx.webredirect.org`)
      - traefik.http.services.wazuhindexer.loadbalancer.server.port=9200
      - traefik.http.routers.wazuhindexer.tls.certresolver=letsencrypt
      - traefik.http.routers.wazuhindexer.tls.domains[0].main=wazuh.xxx.webredirect.org
      - traefik.http.routers.wazuhindexer.tls.domains[0].sans=*.wazuh.xxx.webredirect.org

  wazuh.dashboard:
    image: wazuh/wazuh-dashboard:4.7.1
    hostname: wazuh.dashboard
    restart: always
    networks:
      - proxy
    ports:
      - 5601:5601
    environment:
      - INDEXER_USERNAME=admin
      - INDEXER_PASSWORD=uM0cxxxj7Db%!
      - WAZUH_API_URL=https://wazuh.manager
      - DASHBOARD_USERNAME=kibanaserver
      - DASHBOARD_PASSWORD=kibanaserver
      - API_USERNAME=wazuh-wui
      - API_PASSWORD=MyS3cr37P450r.*-
    volumes:
      - ./config/wazuh_indexer_ssl_certs/wazuh.dashboard.pem:/usr/share/wazuh-dashboard/certs/wazuh-dashboard.pem
      - ./config/wazuh_indexer_ssl_certs/wazuh.dashboard-key.pem:/usr/share/wazuh-dashboard/certs/wazuh-dashboard-key.pem
      - ./config/wazuh_indexer_ssl_certs/root-ca.pem:/usr/share/wazuh-dashboard/certs/root-ca.pem
      - ./config/wazuh_dashboard/opensearch_dashboards.yml:/usr/share/wazuh-dashboard/config/opensearch_dashboards.yml
      - ./config/wazuh_dashboard/wazuh.yml:/usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml
      - wazuh-dashboard-config:/usr/share/wazuh-dashboard/data/wazuh/config
      - wazuh-dashboard-custom:/usr/share/wazuh-dashboard/plugins/wazuh/public/assets/custom
    depends_on:
      - wazuh.indexer
    links:
      - wazuh.indexer:wazuh.indexer
      - wazuh.manager:wazuh.manager
    labels:
      - traefik.enable=true
      - traefik.http.routers.wazuhdash.rule=Host(`wazuh.xxx.webredirect.org`)
      - traefik.http.services.wazuhdash.loadbalancer.server.port=5601
      - traefik.http.routers.wazuhdash.tls.certresolver=letsencrypt
      - traefik.http.routers.wazuhdash.tls.domains[0].main=wazuh.xxx.webredirect.org
      - traefik.http.routers.wazuhdash.tls.domains[0].sans=*.wazuh.xxx.webredirect.org

volumes:
  wazuh_api_configuration:
  wazuh_etc:
  wazuh_logs:
  wazuh_queue:
  wazuh_var_multigroups:
  wazuh_integrations:
  wazuh_active_response:
  wazuh_agentless:
  wazuh_wodles:
  filebeat_etc:
  filebeat_var:
  wazuh-indexer-data:
  wazuh-dashboard-config:
  wazuh-dashboard-custom:

networks:
  proxy:
    external: true
8

Please help!

My config just gets worse day by day.
I'm totally lost

9

Hi. When i have issues of similar sort, i always go for this article

It`s VERY detailed and has a lot of good practices.

Personally, i prefer configuring it via

  labels:
    - "traefik.enable=true"
    ## HTTP Routers
    - "traefik.http.routers.portainer-rtr.entrypoints=https"
    - "traefik.http.routers.portainer-rtr.rule=Host(`portainer.$DOMAINNAME_CLOUD_SERVER`)"
    - "traefik.http.routers.portainer-rtr.tls=true"
    ## Middlewares
    - "traefik.http.routers.portainer-rtr.middlewares=chain-basic-auth@file"
    # - "traefik.http.routers.portainer-rtr.middlewares=chain-no-auth@file"
    ## HTTP Services
    - "traefik.http.routers.portainer-rtr.service=portainer-svc"
    - "traefik.http.services.portainer-svc.loadbalancer.server.port=9000"

Please note that entrypoints setup via CLI for Traefik container ,like this

    command: # CLI arguments
      - --entryPoints.http.address=:80
      - --entryPoints.https.address=:443

My suggestion would be to not do every endpoint at once.
Create an entrypoint. Forget about middlewares for now.
Then configure setup like in lables.

  1. mare sure traefik is enabled
  2. create a router with clear name that this is, indeed, a router. Adding -rtr and -svc (like in the artricle) helped me a lot
  3. define entrypoint for the router; its rule.
  4. map router to a service!
    - "traefik.http.routers.portainer-rtr.service=portainer-svc"
  1. explain the service where traffic should be redirected
    - "traefik.http.services.portainer-svc.loadbalancer.server.port=9000"

Test it. Look trough Traefik logs. Use - --log.level=DEBUG if need be.

Your config is indeed getting messier and its quite difficult to understand what could go wrong where.

Also Traefik Configuration Documentation - Traefik :
"The Static Configuration
There are three different, mutually exclusive (i.e. you can use only one at the same time), ways to define static configuration options in Traefik:

In a configuration file
In the command-line arguments
As environment variables
These ways are evaluated in the order listed above."

So, you may have conflicts in configurations. I`m running 50+ services on multiple independend hosts; services on the same host as Traefik are configured via Lables, which is imho the easiest. Try doing one and poke me with what your logs say when you try to access it.