Nextcloud high performance backend with nginx proxy_pass

hello friends! ive been trying to setup a nextcloud high performance backend based on the open source project. I've got everything working it seems but i have a problem routing traffic through the entrypoints to the service. i've tried so many different variants - it feels like im missing something either very simple or very odd - i'm new to traefik but been bangin my head for days now learning all about it - im a recovering F5 load balancer admin lol - it seems like such and easy setup - maybe i just need another set of eyeballs - im grateful for any help you can provide!

ive deployed nginx and the whole stack into a docker swarm instance- nothing fancy-its all out of the box- i compiled two machines from src but its all working- i can even use wget from inside the traefik instance hitting the nginx instance and it works just fine as expected but no matter how i configure the labels i sometimes get an error 400 http connection to https etc or sometimes a straight up error 500 - it seems the routing through entry point to service is messed up for some reason - from the logs it looks like its not even making it to the nginx webserver. again - im grateful for any help you could give - i know im so close to understanding all of this - im getting there... :wink:

based off of: https:// github. com/strukturag/nextcloud-spreed-signaling/blob/master/docker/docker-compose.yml


traefik.toml:

[global]
  checkNewVersion = true
  SendAnonymousUsage = false

# Enable the Dashboard
[api]
  dashboard = true

[serversTransport]
  insecureSkipVerify = true


# Write out Traefik logs
[log]
  #level = "INFO"
  level = "DEBUG"
  filePath = "/traefik.log"

[entryPoints.http]
  address = ":80"
  # Redirect to HTTPS (why wouldn't you?)
#  [entryPoints.http.http.redirections.entryPoint]
#    to = "https"
#    scheme = "https"

[entryPoints.https]
  address = ":443"
  [entryPoints.https.http.tls]
    #certResolver = "dns-cloudflare"
    certResolver = "main"

# Let's Encrypt
[certificatesResolvers.main.acme]
  email = "xxxx@xxx.com"
  storage = "acme.json"
  # uncomment to use staging CA for testing
  # caServer = "h t t p s : / / acme-staging-v02 . api . letsencrypt . org/directory"
  [certificatesResolvers.main.acme.dnsChallenge]
    provider = "cloudflare"
  # Uncomment to use HTTP validation, like a caveman!
  # [certificatesResolvers.main.acme.httpChallenge]
  #  entryPoint = "http"    


# Docker Traefik provider

[providers.docker]
  endpoint = "unix:///var/run/docker.sock"
  #endpoint = "tcp:///socket-proxy:2375"
  swarmMode = true
  watch = true
  exposedbydefault = false

traefikv2.yml

version: "3.2"

services:
  app:
    image: traefik:v2.4
    env_file: /var/data/config/traefikv2/traefikv2.env
    # Note below that we use host mode to avoid source nat being applied to our ingress HTTP/HTTPS sessions
    # Without host mode, all inbound sessions would have the source IP of the swarm nodes, rather than the
    # original source IP, which would impact logging. If you don't care about this, you can expose ports the 
    # "minimal" way instead
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
      - target: 8080
        published: 8080
        protocol: tcp


    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/data/config/traefikv2/traefik.toml:/traefik.toml:ro
      - /var/data/traefikv2/traefik.log:/traefik.log
      - /var/data/traefikv2/acme.json:/acme.json
      - /var/data/config/traefikv2/conf:/conf
    networks:
      - traefik-public
      - socket-proxy
    # Global mode makes an instance of traefik listen on _every_ node, so that regardless of which
    # node the request arrives on, it'll be forwarded to the correct backend service.
    deploy:
      mode: global
      labels:
        - "traefik.docker.network=traefik-public"
        - "traefik.http.routers.api.rule=Host(`traefik.douno.it`, `traefik.staycuriousandkeepsmil.in`)"
          #- "traefik.http.routers.api.rule=Host(`traefik.douno.it`)"
        - "traefik.http.routers.api.entrypoints=https"
        - "traefik.http.routers.api.tls.domains[0].main=douno.it"
        - "traefik.http.routers.api.tls.domains[0].sans=*.douno.it"        
        - "traefik.http.routers.api.tls.domains[1].main=staycuriousandkeepsmil.in"
        - "traefik.http.routers.api.tls.domains[1].sans=*.staycuriousandkeepsmil.in"        
        - "traefik.http.routers.api.tls=true"
        - "traefik.http.routers.api.tls.certresolver=main"
        - "traefik.http.routers.api.service=api@internal"

          #        - "traefik.http.routers.api.rule=Host(`traefik.staycuriousandkeepsmil.in`)"

        - "traefik.http.services.dummy.loadbalancer.server.port=9999"

        # uncomment this to enable forward authentication on the traefik api/dashboard
        #- "traefik.http.routers.api.middlewares=forward-auth"      
      placement:
        constraints: [node.role == manager]

networks:
  traefik-public:
    external: true
  socket-proxy:
    external: true

docker-compose.yml

version: '3'

networks:
  spreed:
    external: true
  traefik-public:
    external: true

services:

  nginx:
    image: lscr.io/linuxserver/nginx
    container_name: nextcloud_spreed_nginx
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=America/New_York
    volumes:
      - ./nginx/conf:/config
      - ./certbot/conf/:/etc/nginx/ssl/:ro
    networks:
      - spreed
      - traefik-public
   # ports:
   #   - 80:80
   #   - 443:443
    deploy:
      labels:
        - traefik.enable=true
        - traefik.docker.network=traefik-public

#        - traefik.http.routers.ng-http.rule=Host(`signaling.local.douno.it`)
#        - traefik.http.routers.ng-http.entrypoints=http
#        - traefik.http.routers.ng-http.service=ng
#        - traefik.http.services.ng.loadbalancer.server.scheme=http
#        - traefik.http.routers.ng-https.rule=Host(`signaling.local.xxx.xxx`)
#        - traefik.http.routers.ng-https.entrypoints=https
#        - traefik.http.routers.ng-https.tls=true
#        - traefik.http.routers.ng-https.tls.certresolver=main
#        - traefik.http.routers.ng-https.service=ng
        # - traefik.http.services.ng.loadbalancer.server.port=443
#        - traefik.http.services.ng.loadbalancer.server.scheme=https
#        - traefik.http.services.ng.loadbalancer.passHostHeader=true
        - "traefik.http.routers.nginx-hpb-rtr-insecure.rule=Host(`signaling.douno.it`)"
        - "traefik.http.routers.nginx-hpb-rtr-insecure.entrypoints=http"
        - "traefik.http.routers.nginx-hpb-rtr-insecure.service=nginx-hpb-svc-insecure"
        - "traefik.http.services.nginx-hpb-svc-insecure.loadbalancer.server.port=80"
        - "traefik.http.services.nginx-hpb-svc-insecure.loadbalancer.server.scheme=http"
        - "traefik.http.services.nginx-hpb-svc-insecure.loadbalancer.passHostHeader=true"


#                              .must match router below.
#        - "traefik.http.routers.ng.rule=Host(`signal.local.xxx.xxx`)"
#                               can-b-anything              .services match.
#        - "traefik.http.routers.ng.service=ng"
#                                .services match.
#        - "traefik.http.services.ng.loadbalancer.server.port=80"
#        - traefik.http.services.ng.loadbalancer.server.scheme=http
#        - "traefik.http.services.ng.loadbalancer.passHostHeader=true"

        - "traefik.http.routers.nginx-hpb-rtr-secure.rule=Host(`signaling.xxx.xxx`)"
        - "traefik.http.routers.nginx-hpb-rtr-secure.entrypoints=https"
        - "traefik.http.routers.nginx-hpb-rtr-secure.service=nginx-hpb-svc-secure"
        - "traefik.http.services.nginx-hpb-svc-secure.loadbalancer.server.port=443"
        - "traefik.http.services.nginx-hpb-svc-secure.loadbalancer.server.scheme=https"
        - "traefik.http.services.nginx-hpb-svc-secure.loadbalancer.passHostHeader=true"


#        - "traefik.http.routers.ng.rule=Host(`signaling.local.xxx.xxx`)"
#        - "traefik.http.routers.ng.entrypoints=https"
#        - "traefik.http.routers.ng.tls.domains[0].main=xxx.xxx"
#        - "traefik.http.routers.ng.tls.domains[0].sans=*.xxx.xxx"        
#        - "traefik.http.routers.ng.tls=true"
#        - "traefik.http.routers.ng.tls.certresolver=main"
#        - "traefik.http.routers.ng.service=api@internal"
#        - "traefik.http.routers.ng.service=ng"
#        - "traefik.http.services.ng.loadbalancer.server.port=443"
#        - "traefik.http.services.ng.loadbalancer.passHostHeader=true"

    restart: unless-stopped
    depends_on:
      - spreedbackend
      - nats
      - janus
      - coturn

  certbot:
    image: certbot/certbot:latest
    container_name: nextcloud_spreed_certbot
    networks:
      - spreed
    volumes:
      - ./nginx/conf/www:/var/www/certbot/:rw
      - ./certbot/conf/:/etc/letsencrypt/:rw
    depends_on:
      - nginx


  spreedbackend:
    image: nextcloud-spreed-signaling_spreedbackend

#    build:
#      context: .
#      dockerfile: docker/server/Dockerfile
    volumes:
      - ./server.conf:/config/server.conf
    #network_mode: host
    networks:
      - spreed
    #ports:
      #- 8080:8080
    container_name: nextcloud_spreed_backend
    logging:
      options:
        max-size: 10m
    restart: unless-stopped
    depends_on:
      - nats
      - janus
      - coturn

  nats:
    image: nats:2.2.1
    volumes:
      - ./gnatsd.conf:/config/gnatsd.conf
    command: ["-c", "/config/gnatsd.conf"]
    #network_mode: host
    networks:
      - spreed
    container_name: nextcloud_spreed_nats
    logging:
      options:
        max-size: 10m
    restart: unless-stopped

  janus:
    image: nextcloud-spreed-signaling_janus
    #build: docker/janus
    command: ["janus", "--full-trickle"]
    #network_mode: host
    networks:
      - spreed
    container_name: nextcloud_spreed_janus
    logging:
      options:
        max-size: 10m
    restart: unless-stopped

  coturn:
    image: coturn/coturn:latest
    #network_mode: host
    networks:
      - spreed
    ports:
      - "3478:3478/tcp"
      - "3478:3478/udp"
    container_name: nextcloud_spreed_coturn
    logging:
      options:
        max-size: 10m
    command:
      - "--realm"
      - "${SIGNAL_DOMAIN}"
      - "--static-auth-secret"
      - "${STATIC_SECRET}"
      - "--no-stdout-log"
      - "--log-file"
      - "stdout"
      - "--stale-nonce=600"
      - "--use-auth-secret"
      - "--lt-cred-mech"
      - "--fingerprint"
      - "--no-software-attribute"
      - "--no-multicast-peers"
    restart: unless-stopped

nginx conf:

upstream signaling {
    server spreedbackend:8080;
}

server {
    server_name signaling.xxx.xxx;

    listen 443 ssl; # managed by Certbot
    root /config/www;
    index index.html index.htm index.php;
    #ssl_certificate /config/keys/cert.crt;
    #ssl_certificate_key /config/keys/cert.key;
    ssl_certificate /etc/nginx/ssl/dummy/signaling.xxx.xxx/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/nginx/ssl/dummy/signaling.xxx.xxx/privkey.pem; # managed by Certbot
    #include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    #ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
    #add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";

location /standalone-signaling/ {
        proxy_pass http://signaling/;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

    location /standalone-signaling/spreed {
        proxy_pass http://signaling/spreed;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

}
server {
    if ($host = signaling.xxx.xxx) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    listen 80;
    server_name signaling.xxx.xxx;
    return 404; # managed by Certbot


}

i hope to get this behind me - i have tons more configs to work on migrating! :slight_smile:

-chefboyrdave2.1 aka daveK

How do you install/run nextcloud? With all-in-one? You use docker stack deploy? (In Swarm depends_on does not work.)

Why do you use a very old Traefik version? You want to use Traefik Configuration Discovery via labels or use a manual "dynamic config" file?

What's the plan for TLS certs, who creates them, who uses them?

1 Like

I was using the official apache nextcloud image - i prefer to have more control over the components. I actually finally got it working - yeah - do use stack deploy - i am using an env file so i have to use docker-compose to generate the file :slight_smile: - this is what im running specifically for my apps - cd /var/data/config/ncloud && docker stack deploy -c <(docker-compose config|grep -v name) ncloud && cd /var/data

im using the dynamic labels and it seems to be working now - i had to go through a redeployment of traefik with a fine tooth comb- i think i had a couple issues 1) it was automagically creating the endpoints for me based on the docker containers and 2) i didnt have the router svc line linking the svc to the route/endpoints - i seem to have better luck when i specify each entry in detail prescriptively rather than relying on traefik to do it for me - i like to understand how its gonna work. I ended up putting a constraint on my app now too - so they must match the label for the provider to use it - otherwise it's filtered out and not used -gives me even more control and keeps from apps being exposed by accident while i dont really know what im doing :man_facepalming: :rofl: :v:

good point re: the old version - i'll update that now as well- thanks for noticing! :pray: :v: