Nextcloud high performance backend with nginx proxy_pass

chefboyrdave2.1 · February 1, 2024, 3:35pm

hello friends! ive been trying to setup a nextcloud high performance backend based on the open source project. I've got everything working it seems but i have a problem routing traffic through the entrypoints to the service. i've tried so many different variants - it feels like im missing something either very simple or very odd - i'm new to traefik but been bangin my head for days now learning all about it - im a recovering F5 load balancer admin lol - it seems like such and easy setup - maybe i just need another set of eyeballs - im grateful for any help you can provide!

ive deployed nginx and the whole stack into a docker swarm instance- nothing fancy-its all out of the box- i compiled two machines from src but its all working- i can even use wget from inside the traefik instance hitting the nginx instance and it works just fine as expected but no matter how i configure the labels i sometimes get an error 400 http connection to https etc or sometimes a straight up error 500 - it seems the routing through entry point to service is messed up for some reason - from the logs it looks like its not even making it to the nginx webserver. again - im grateful for any help you could give - i know im so close to understanding all of this - im getting there...

based off of: https:// github. com/strukturag/nextcloud-spreed-signaling/blob/master/docker/docker-compose.yml

traefik.toml:

[global]
  checkNewVersion = true
  SendAnonymousUsage = false

# Enable the Dashboard
[api]
  dashboard = true

[serversTransport]
  insecureSkipVerify = true


# Write out Traefik logs
[log]
  #level = "INFO"
  level = "DEBUG"
  filePath = "/traefik.log"

[entryPoints.http]
  address = ":80"
  # Redirect to HTTPS (why wouldn't you?)
#  [entryPoints.http.http.redirections.entryPoint]
#    to = "https"
#    scheme = "https"

[entryPoints.https]
  address = ":443"
  [entryPoints.https.http.tls]
    #certResolver = "dns-cloudflare"
    certResolver = "main"

# Let's Encrypt
[certificatesResolvers.main.acme]
  email = "xxxx@xxx.com"
  storage = "acme.json"
  # uncomment to use staging CA for testing
  # caServer = "h t t p s : / / acme-staging-v02 . api . letsencrypt . org/directory"
  [certificatesResolvers.main.acme.dnsChallenge]
    provider = "cloudflare"
  # Uncomment to use HTTP validation, like a caveman!
  # [certificatesResolvers.main.acme.httpChallenge]
  #  entryPoint = "http"    


# Docker Traefik provider

[providers.docker]
  endpoint = "unix:///var/run/docker.sock"
  #endpoint = "tcp:///socket-proxy:2375"
  swarmMode = true
  watch = true
  exposedbydefault = false

traefikv2.yml

version: "3.2"

services:
  app:
    image: traefik:v2.4
    env_file: /var/data/config/traefikv2/traefikv2.env
    # Note below that we use host mode to avoid source nat being applied to our ingress HTTP/HTTPS sessions
    # Without host mode, all inbound sessions would have the source IP of the swarm nodes, rather than the
    # original source IP, which would impact logging. If you don't care about this, you can expose ports the 
    # "minimal" way instead
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
      - target: 8080
        published: 8080
        protocol: tcp


    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/data/config/traefikv2/traefik.toml:/traefik.toml:ro
      - /var/data/traefikv2/traefik.log:/traefik.log
      - /var/data/traefikv2/acme.json:/acme.json
      - /var/data/config/traefikv2/conf:/conf
    networks:
      - traefik-public
      - socket-proxy
    # Global mode makes an instance of traefik listen on _every_ node, so that regardless of which
    # node the request arrives on, it'll be forwarded to the correct backend service.
    deploy:
      mode: global
      labels:
        - "traefik.docker.network=traefik-public"
        - "traefik.http.routers.api.rule=Host(`traefik.douno.it`, `traefik.staycuriousandkeepsmil.in`)"
          #- "traefik.http.routers.api.rule=Host(`traefik.douno.it`)"
        - "traefik.http.routers.api.entrypoints=https"
        - "traefik.http.routers.api.tls.domains[0].main=douno.it"
        - "traefik.http.routers.api.tls.domains[0].sans=*.douno.it"        
        - "traefik.http.routers.api.tls.domains[1].main=staycuriousandkeepsmil.in"
        - "traefik.http.routers.api.tls.domains[1].sans=*.staycuriousandkeepsmil.in"        
        - "traefik.http.routers.api.tls=true"
        - "traefik.http.routers.api.tls.certresolver=main"
        - "traefik.http.routers.api.service=api@internal"

          #        - "traefik.http.routers.api.rule=Host(`traefik.staycuriousandkeepsmil.in`)"

        - "traefik.http.services.dummy.loadbalancer.server.port=9999"

        # uncomment this to enable forward authentication on the traefik api/dashboard
        #- "traefik.http.routers.api.middlewares=forward-auth"      
      placement:
        constraints: [node.role == manager]

networks:
  traefik-public:
    external: true
  socket-proxy:
    external: true

docker-compose.yml

version: '3'

networks:
  spreed:
    external: true
  traefik-public:
    external: true

services:

  nginx:
    image: lscr.io/linuxserver/nginx
    container_name: nextcloud_spreed_nginx
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=America/New_York
    volumes:
      - ./nginx/conf:/config
      - ./certbot/conf/:/etc/nginx/ssl/:ro
    networks:
      - spreed
      - traefik-public
   # ports:
   #   - 80:80
   #   - 443:443
    deploy:
      labels:
        - traefik.enable=true
        - traefik.docker.network=traefik-public

#        - traefik.http.routers.ng-http.rule=Host(`signaling.local.douno.it`)
#        - traefik.http.routers.ng-http.entrypoints=http
#        - traefik.http.routers.ng-http.service=ng
#        - traefik.http.services.ng.loadbalancer.server.scheme=http
#        - traefik.http.routers.ng-https.rule=Host(`signaling.local.xxx.xxx`)
#        - traefik.http.routers.ng-https.entrypoints=https
#        - traefik.http.routers.ng-https.tls=true
#        - traefik.http.routers.ng-https.tls.certresolver=main
#        - traefik.http.routers.ng-https.service=ng
        # - traefik.http.services.ng.loadbalancer.server.port=443
#        - traefik.http.services.ng.loadbalancer.server.scheme=https
#        - traefik.http.services.ng.loadbalancer.passHostHeader=true
        - "traefik.http.routers.nginx-hpb-rtr-insecure.rule=Host(`signaling.douno.it`)"
        - "traefik.http.routers.nginx-hpb-rtr-insecure.entrypoints=http"
        - "traefik.http.routers.nginx-hpb-rtr-insecure.service=nginx-hpb-svc-insecure"
        - "traefik.http.services.nginx-hpb-svc-insecure.loadbalancer.server.port=80"
        - "traefik.http.services.nginx-hpb-svc-insecure.loadbalancer.server.scheme=http"
        - "traefik.http.services.nginx-hpb-svc-insecure.loadbalancer.passHostHeader=true"


#                              .must match router below.
#        - "traefik.http.routers.ng.rule=Host(`signal.local.xxx.xxx`)"
#                               can-b-anything              .services match.
#        - "traefik.http.routers.ng.service=ng"
#                                .services match.
#        - "traefik.http.services.ng.loadbalancer.server.port=80"
#        - traefik.http.services.ng.loadbalancer.server.scheme=http
#        - "traefik.http.services.ng.loadbalancer.passHostHeader=true"

        - "traefik.http.routers.nginx-hpb-rtr-secure.rule=Host(`signaling.xxx.xxx`)"
        - "traefik.http.routers.nginx-hpb-rtr-secure.entrypoints=https"
        - "traefik.http.routers.nginx-hpb-rtr-secure.service=nginx-hpb-svc-secure"
        - "traefik.http.services.nginx-hpb-svc-secure.loadbalancer.server.port=443"
        - "traefik.http.services.nginx-hpb-svc-secure.loadbalancer.server.scheme=https"
        - "traefik.http.services.nginx-hpb-svc-secure.loadbalancer.passHostHeader=true"


#        - "traefik.http.routers.ng.rule=Host(`signaling.local.xxx.xxx`)"
#        - "traefik.http.routers.ng.entrypoints=https"
#        - "traefik.http.routers.ng.tls.domains[0].main=xxx.xxx"
#        - "traefik.http.routers.ng.tls.domains[0].sans=*.xxx.xxx"        
#        - "traefik.http.routers.ng.tls=true"
#        - "traefik.http.routers.ng.tls.certresolver=main"
#        - "traefik.http.routers.ng.service=api@internal"
#        - "traefik.http.routers.ng.service=ng"
#        - "traefik.http.services.ng.loadbalancer.server.port=443"
#        - "traefik.http.services.ng.loadbalancer.passHostHeader=true"

    restart: unless-stopped
    depends_on:
      - spreedbackend
      - nats
      - janus
      - coturn

  certbot:
    image: certbot/certbot:latest
    container_name: nextcloud_spreed_certbot
    networks:
      - spreed
    volumes:
      - ./nginx/conf/www:/var/www/certbot/:rw
      - ./certbot/conf/:/etc/letsencrypt/:rw
    depends_on:
      - nginx


  spreedbackend:
    image: nextcloud-spreed-signaling_spreedbackend

#    build:
#      context: .
#      dockerfile: docker/server/Dockerfile
    volumes:
      - ./server.conf:/config/server.conf
    #network_mode: host
    networks:
      - spreed
    #ports:
      #- 8080:8080
    container_name: nextcloud_spreed_backend
    logging:
      options:
        max-size: 10m
    restart: unless-stopped
    depends_on:
      - nats
      - janus
      - coturn

  nats:
    image: nats:2.2.1
    volumes:
      - ./gnatsd.conf:/config/gnatsd.conf
    command: ["-c", "/config/gnatsd.conf"]
    #network_mode: host
    networks:
      - spreed
    container_name: nextcloud_spreed_nats
    logging:
      options:
        max-size: 10m
    restart: unless-stopped

  janus:
    image: nextcloud-spreed-signaling_janus
    #build: docker/janus
    command: ["janus", "--full-trickle"]
    #network_mode: host
    networks:
      - spreed
    container_name: nextcloud_spreed_janus
    logging:
      options:
        max-size: 10m
    restart: unless-stopped

  coturn:
    image: coturn/coturn:latest
    #network_mode: host
    networks:
      - spreed
    ports:
      - "3478:3478/tcp"
      - "3478:3478/udp"
    container_name: nextcloud_spreed_coturn
    logging:
      options:
        max-size: 10m
    command:
      - "--realm"
      - "${SIGNAL_DOMAIN}"
      - "--static-auth-secret"
      - "${STATIC_SECRET}"
      - "--no-stdout-log"
      - "--log-file"
      - "stdout"
      - "--stale-nonce=600"
      - "--use-auth-secret"
      - "--lt-cred-mech"
      - "--fingerprint"
      - "--no-software-attribute"
      - "--no-multicast-peers"
    restart: unless-stopped

nginx conf:

upstream signaling {
    server spreedbackend:8080;
}

server {
    server_name signaling.xxx.xxx;

    listen 443 ssl; # managed by Certbot
    root /config/www;
    index index.html index.htm index.php;
    #ssl_certificate /config/keys/cert.crt;
    #ssl_certificate_key /config/keys/cert.key;
    ssl_certificate /etc/nginx/ssl/dummy/signaling.xxx.xxx/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/nginx/ssl/dummy/signaling.xxx.xxx/privkey.pem; # managed by Certbot
    #include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    #ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
    #add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";

location /standalone-signaling/ {
        proxy_pass http://signaling/;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

    location /standalone-signaling/spreed {
        proxy_pass http://signaling/spreed;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

}
server {
    if ($host = signaling.xxx.xxx) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    listen 80;
    server_name signaling.xxx.xxx;
    return 404; # managed by Certbot


}

i hope to get this behind me - i have tons more configs to work on migrating!

-chefboyrdave2.1 aka daveK

bluepuma77 · February 2, 2024, 10:48pm

How do you install/run nextcloud? With all-in-one? You use docker stack deploy? (In Swarm depends_on does not work.)

Why do you use a very old Traefik version? You want to use Traefik Configuration Discovery via labels or use a manual "dynamic config" file?

What's the plan for TLS certs, who creates them, who uses them?

chefboyrdave2.1 · February 3, 2024, 6:32pm

I was using the official apache nextcloud image - i prefer to have more control over the components. I actually finally got it working - yeah - do use stack deploy - i am using an env file so i have to use docker-compose to generate the file - this is what im running specifically for my apps - cd /var/data/config/ncloud && docker stack deploy -c <(docker-compose config|grep -v name) ncloud && cd /var/data

im using the dynamic labels and it seems to be working now - i had to go through a redeployment of traefik with a fine tooth comb- i think i had a couple issues 1) it was automagically creating the endpoints for me based on the docker containers and 2) i didnt have the router svc line linking the svc to the route/endpoints - i seem to have better luck when i specify each entry in detail prescriptively rather than relying on traefik to do it for me - i like to understand how its gonna work. I ended up putting a constraint on my app now too - so they must match the label for the provider to use it - otherwise it's filtered out and not used -gives me even more control and keeps from apps being exposed by accident while i dont really know what im doing

chefboyrdave2.1 · February 3, 2024, 6:34pm

good point re: the old version - i'll update that now as well- thanks for noticing!

chefboyrdave2.1 · March 4, 2024, 6:59pm

ok - i'm back at it now- i have more experience under my belt but less hair and still no solution:

https://github.com/strukturag/nextcloud-spreed-signaling/blob/master/README.md

Caddy

v1

Caddy (v1) configuration:

myserver.domain.invalid {
  proxy /standalone-signaling/ http://127.0.0.1:8080 {
    without /standalone-signaling
    transparent
    websocket
  }
}

v2

Caddy (v2) configuration:

myserver.domain.invalid {
  route /standalone-signaling/* {
    uri strip_prefix /standalone-signaling
    reverse_proxy http://127.0.0.1:8080
  }
}

i am able to get the HPB working behind nginx in a simple docker stack. this is progress but i need to deploy this inside my traefik + docker swarm cluster. i've tried deploying the whole stack with nginx behind traefik and i get a 502- ideally id like to drop nginx and go directly to the spreedbackend:8080 - when i try with the following config i get a 404 when going directly to the spreedbackend:8080:

snip from my docker-compose stack:

  spreedbackend:
    image: localhost:5000/nextcloud-spreed-signaling_spreedbackend
    #image: strukturag/nextcloud-spreed-signaling
    #image: spreed-signaling_spreedbackend
    env_file: /var/data/config/spreed-signaling/spreed-signaling.env
    deploy:
      replicas: 1
      placement:
        constraints:
          - "node.labels.node==node1"
      labels:
        - "traefik.enable=true"
        - "traefik.docker.network=cloud-public"
        - "traefik.docker.lbswarm=true"
        - "traefik.constraint-label=cloud-public"

        # traefikv2
        - "traefik.http.routers.spreed-signaling-secure.entrypoints=https"
        - "traefik.http.routers.spreed-signaling-secure.rule=Host(`signaling.any.tld`)
        - "traefik.http.routers.spreed-signaling-secure.tls=true"
        - "traefik.http.routers.spreed-signaling-secure.tls.certresolver=main"
        - "traefik.http.routers.spreed-signaling-secure.service=spreed-signaling-secure"
        - "traefik.http.services.spreed-signaling-secure.loadbalancer.server.port=8080"


    volumes:
      - /var/data/spreed-signaling/spreedbackend/server.conf:/config/server.conf

    networks:
      - spreed-signaling
      - cloud-public

from external public internet --> traefik spreedbackend 443 router:

wget  https://signaling.some.url.tld/standalone-signaling/api
/v1/welcome
--2024-03-04 13:55:13--  https://signaling.some.url.tld/standalone-signaling/api/v1/welcome
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving signaling.some.url.tld (signaling.some.url.tld)... 209.59.156.102
Connecting to signaling.some.url.tld (signaling.some.url.tld)|209.59.156.102|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2024-03-04 13:55:13 ERROR 404: Not Found.

from the traefik container --> spreedbackend container http://172.16.200.39:8080

wget http://172.16.200.39:8080/standalone-signaling/api/v1/welcome
Connecting to 172.16.200.39:8080 (172.16.200.39:8080)
wget: server returned error: HTTP/1.1 404 Not Found

you can see i a simple caddy config - it looks like a simple proxy pass or websocket etc. i need help to convert to a working config for traefik- i'm not confident enough yet in my ability to create a working config for this- ive been troubleshooting different things for quite awhile - getting the stack up n runnin was working in itself! - thanks so much for your help friend- i hope your having the best day!

-D

chefboyrdave2.1 · March 4, 2024, 7:04pm

after further learning about the orders of docker environment variable hierarchy- i now use this to start my stacks:

export APPNAME=spreed-signaling

export $(cat /var/data/config/${APPNAME}/$APPNAME.sh.env) > /dev/null 2>&1; docker stack deploy -c /var/data/config/${APPNAME}/$APPNAME.yml ${APPNAME}

inside my $APPNAME.yml which is essentially the docker-compose.yaml file - i have an env_file that points to /var/data/config/${APPNAME}/$APPNAME.env - thus giving me the ability to modify the variables of the docker-compose.yaml file and also pass env variables to the container on startup- lost a few hairs until this realization

bluepuma77 · March 4, 2024, 8:36pm

should probably look something like this, placed on the target service/container:

labels:
  - traefik.http.routers.router1.rule=Host(`myserver.domain.invalid`) && PathPrefix(`/standalone-signaling/`)
  - traefik.http.routers.router1.middlewares=myStrip
  - traefik.http.middlewares.myStrip.stripprefix.prefixes=/standalone-signaling
  - traefik.http.services.router1.loadbalancer.server.port=8080

Note this uses the container IP as target, not localhost, should be available within a Docker network.

chefboyrdave2.1 · March 5, 2024, 3:24pm

tyvm- this looks exactly what i was looking for! in the meantime i configured their vanilla setup- the nginx webserver in the equation since i know how to configure traefik for that - i'm still not convinced my spreedbackend:8080 is listening / working properly- allegedly someone has the stack working with a standalone docker setup- i converted it to a docker swarm setup but i can't get the listener on the container to come up... i plan on posting my configs once i get this working- in hopes of saving someone else a weekend and days!

-D

bluepuma77 · March 5, 2024, 4:00pm

Swarm needs labels under deploy section and you must declare the target port to be used.

Check simple Traefik Swarm example.

chefboyrdave2.1 · March 6, 2024, 11:43am

ah- good idea - i run all my apps behind traefik as a reverse proxy on the edge- i'll try swappin out the nginx on the standalone instance as a next intermediary step use it as an internal lb- i was having a heck of a time trying to get the standalone docker-compose.yaml stack working and tested- I think i made good progress towards a working config- ive documented my testing here: More detailed information about docker compose · Issue #411 · strukturag/nextcloud-spreed-signaling · GitHub

Topic		Replies	Views
Traefik2 + Nextcloud = tons of errors Traefik v2 docker	1	629	February 11, 2024
Traefik2 Local Services Traefik v2 docker-swarm	5	1295	May 3, 2020
Really struggling: trying to set up non-docker on same device as Traefik, to go through Traefik Traefik v2 file	6	1862	May 19, 2020
Bad Gateway - Run out of ideas! (Newbie) Traefik v2 docker	1	476	March 31, 2020
Nextcloud / Traefik uncooperative (redirecting to https://_/) Traefik v1 docker	2	2808	July 1, 2021