How to fix ACME error 400 connection refused

Can't seem to figure out why I am having this unable to obtain ACME error 400 connection refused. Any insight from anyone would be greatly appreciated.

My docker-compose.yml

version: '3.7'

services:

  traefik:
    image: traefik:2.3.2
    ports:
      - 80:80
      - 443:443
    deploy:
      resources:
        limits:
          cpus: '0.50'
          memory: 512M
        reservations:
          cpus: '0.25'
          memory: 256M
      placement:
        constraints:
          # Make the traefik service run only on the node with this label
          # as the node with it has the volume for the certificates
          - node.labels.reverse-proxy.reverse-proxy-certificates == true
          - node.role == manager
      labels:
        - constraint-label=reverse-proxy

        - traefik.enable=true
        - traefik.docker.network=reverse-proxy
        
        - traefik.http.middlewares.admin-auth.basicauth.users=${USERNAME}:${HASHED_PASSWORD}
        
        - traefik.http.routers.reverse-proxy-https.rule=Host(`${DOMAIN}`)
        - traefik.http.routers.reverse-proxy-https.entrypoints=web,web-secure
        - traefik.http.routers.reverse-proxy-https.tls=true
        - traefik.http.routers.reverse-proxy-https.service=api@internal
        - traefik.http.routers.reverse-proxy-https.tls.certresolver=le
        - traefik.http.routers.reverse-proxy-https.middlewares=admin-auth
        - traefik.http.routers.reverse-proxy.tls.domains[0].main=DOMAIN.TLD
        - traefik.http.routers.reverse-proxy.tls.domains[0].sans=*.DOMAIN.TLD
      
        - traefik.http.services.reverse-proxy.loadbalancer.server.port=8080

    volumes:
      # Add Docker as a mounted volume, so that Traefik can read the labels of other services
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      # Mount the volume to store the certificates
      - "/srv/traefik.DOMAIN.TLD/certs:/certificates"
      - "/srv/traefik.DOMAIN.TLD/letsencrypt:/letsencrypt"
      #- "reverse-proxy-certificates:/certificates"
      #- "reverse-proxy-certificates:/letsencrypt"
    command:
      - --accesslog
      
      - --log
      - --log.level=DEBUG

      - --api

      - --providers.docker
      - --providers.docker.constraints=Label(`constraint-label`, `reverse-proxy`)
      - --providers.docker.exposedbydefault=false
      - --providers.docker.swarmmode
      
      - --entrypoints.web.address=:80
      # global redirect to https
      - --entrypoints.web.http.redirections.entryPoint.to=web-secure
      - --entrypoints.web.http.redirections.entrypoint.scheme=https
      - --entrypoints.web-secure.address=:443
    
      - --certificatesresolvers.le.acme.email=${EMAIL}
      - --certificatesresolvers.le.acme.storage=/letsencrypt/acme.json
      - --certificatesresolvers.le.acme.tlschallenge=true
      #- --certificatesresolvers.le.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
    networks:
      # Use the public network created to be shared between Traefik and
      # any other service that needs to be publicly available with HTTPS
      - reverse-proxy
    extra_hosts:
      - EXTRA_HOSTS
    environment:
      - NODE_ID=5431vdsvsdcsdcsdcdsc
      - EMAIL=admin@DOMAIN.TLD
      
      - DOMAIN=traefik.DOMAIN.TLD
      - USERNAME=admin
      - PASSWORD="changethis"
      - HASHED_PASSWORD='$$apr1$$6XaPhlNf$$N.NgCREOAXsVNmV8IKk//w.'
      #- HASHPASS_ENV: $(HASHPASS_HOST:~(openssl passwd -apr1 $PASSWORD)) 
      #- $SRV_VPATH="/srv/DOMAIN.TLD/"
      - EXTRA_HOSTS="traefik.DOMAIN.TLD:123.456.789.123"
    env_file:
     - ./.env

  whoami:
    image: "traefik/whoami"
    container_name: "simple-service"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`whoami.DOMAIN.TLD`)"
      - "traefik.http.routers.whoami.middlewares=auth"
      - "traefik.http.routers.whoami.entrypoints=web,web-secure"
      - "traefik.http.routers.whoami.tls.certresolver=le"
    networks:
      - reverse-proxy

volumes:
  # Create a volume to store the certificates, there is a constraint to make sure
  # Traefik is always deployed to the same Docker node with the same volume containing
  # the HTTPS certificates
  reverse-proxy-certificates:
networks:
  # Use the previously created public network "reverse-proxy", shared with other
  # services that need to be publicly available via this Traefik
  reverse-proxy:
    external: true
    driver: overlay

My error log

traefik_traefik.1.jw56pnnuxasqgsbhxhedpjp5d@ninjaserver    | time="2020-11-10T04:38:03Z" level=error msg="Unable to obtain ACME certificate for domains \"traefik.whiskeyonthe.rocks\": unable to generate a certificate for the domains [traefik.whiskeyonthe.rocks]: error: one or more domains had a problem:\n[traefik.whiskeyonthe.rocks] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Connection refused, url: \n" providerName=le.acme routerName=reverse-proxy-https@docker rule="Host(`traefik.whiskeyonthe.rocks`)"

Thanks for your insight its greatly appreciated :slight_smile:

Check your firewalls and any port forwards are correct. Connection is refused.

https://letsdebug.net/ can assist with debugging. But it gives the same error in this case.

1 Like

Hello I have check my firewall and portforwards, everything seems to be correct. I even reset my IP tables and still having same issue. I can't seem to find why the ports are refusing connection even though they are open, exposed and listening.

Firewall

ronin@ninjaserver:~$ sudo ufw status numbered
[sudo] password for ronin: 
Status: active
     To                         Action      From
     --                         ------      ----
[ 1] 4789/udp                   ALLOW IN    Anywhere                  
[ 2] 7946                       ALLOW IN    Anywhere                  
[ 3] 2377/tcp                   ALLOW IN    Anywhere                  
[ 4] 22                         ALLOW IN    Anywhere                  
[ 5] 80                         ALLOW IN    Anywhere                  
[ 6] 443                        ALLOW IN    Anywhere                  
[ 7] 4789/udp (v6)              ALLOW IN    Anywhere (v6)             
[ 8] 7946 (v6)                  ALLOW IN    Anywhere (v6)             
[ 9] 2377/tcp (v6)              ALLOW IN    Anywhere (v6)             
[10] 22 (v6)                    ALLOW IN    Anywhere (v6)             
[11] 80 (v6)                    ALLOW IN    Anywhere (v6)             
[12] 443 (v6)                   ALLOW IN    Anywhere (v6) 

IPtables

ronin@ninjaserver:~$ sudo iptables -L
Chain INPUT (policy DROP)
target     prot opt source               destination         
ufw-before-logging-input  all  --  anywhere             anywhere            
ufw-before-input  all  --  anywhere             anywhere            
ufw-after-input  all  --  anywhere             anywhere            
ufw-after-logging-input  all  --  anywhere             anywhere            
ufw-reject-input  all  --  anywhere             anywhere            
ufw-track-input  all  --  anywhere             anywhere            
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-INGRESS  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ufw-before-logging-forward  all  --  anywhere             anywhere            
ufw-before-forward  all  --  anywhere             anywhere            
ufw-after-forward  all  --  anywhere             anywhere            
ufw-after-logging-forward  all  --  anywhere             anywhere            
ufw-reject-forward  all  --  anywhere             anywhere            
ufw-track-forward  all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ufw-before-logging-output  all  --  anywhere             anywhere            
ufw-before-output  all  --  anywhere             anywhere            
ufw-after-output  all  --  anywhere             anywhere            
ufw-after-logging-output  all  --  anywhere             anywhere            
ufw-reject-output  all  --  anywhere             anywhere            
ufw-track-output  all  --  anywhere             anywhere            

Chain DOCKER (2 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             172.19.0.5           tcp dpt:9001

Chain DOCKER-INGRESS (1 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https
ACCEPT     tcp  --  anywhere             anywhere             state RELATED,ESTABLISHED tcp spt:https
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere             state RELATED,ESTABLISHED tcp spt:http
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain ufw-after-forward (1 references)
target     prot opt source               destination         

Chain ufw-after-input (1 references)
target     prot opt source               destination         
ufw-skip-to-policy-input  udp  --  anywhere             anywhere             udp dpt:netbios-ns
ufw-skip-to-policy-input  udp  --  anywhere             anywhere             udp dpt:netbios-dgm
ufw-skip-to-policy-input  tcp  --  anywhere             anywhere             tcp dpt:netbios-ssn
ufw-skip-to-policy-input  tcp  --  anywhere             anywhere             tcp dpt:microsoft-ds
ufw-skip-to-policy-input  udp  --  anywhere             anywhere             udp dpt:bootps
ufw-skip-to-policy-input  udp  --  anywhere             anywhere             udp dpt:bootpc
ufw-skip-to-policy-input  all  --  anywhere             anywhere             ADDRTYPE match dst-type BROADCAST

Chain ufw-after-logging-forward (1 references)
target     prot opt source               destination         
LOG        all  --  anywhere             anywhere             limit: avg 3/min burst 10 LOG level warning prefix "[UFW BLOCK] "

Chain ufw-after-logging-input (1 references)
target     prot opt source               destination         
LOG        all  --  anywhere             anywhere             limit: avg 3/min burst 10 LOG level warning prefix "[UFW BLOCK] "

Chain ufw-after-logging-output (1 references)
target     prot opt source               destination         

Chain ufw-after-output (1 references)
target     prot opt source               destination         

Chain ufw-before-forward (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere             icmp destination-unreachable
ACCEPT     icmp --  anywhere             anywhere             icmp time-exceeded
ACCEPT     icmp --  anywhere             anywhere             icmp parameter-problem
ACCEPT     icmp --  anywhere             anywhere             icmp echo-request
ufw-user-forward  all  --  anywhere             anywhere            

Chain ufw-before-input (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ufw-logging-deny  all  --  anywhere             anywhere             ctstate INVALID
DROP       all  --  anywhere             anywhere             ctstate INVALID
ACCEPT     icmp --  anywhere             anywhere             icmp destination-unreachable
ACCEPT     icmp --  anywhere             anywhere             icmp time-exceeded
ACCEPT     icmp --  anywhere             anywhere             icmp parameter-problem
ACCEPT     icmp --  anywhere             anywhere             icmp echo-request
ACCEPT     udp  --  anywhere             anywhere             udp spt:bootps dpt:bootpc
ufw-not-local  all  --  anywhere             anywhere            
ACCEPT     udp  --  anywhere             224.0.0.251          udp dpt:mdns
ACCEPT     udp  --  anywhere             239.255.255.250      udp dpt:1900
ufw-user-input  all  --  anywhere             anywhere            

Chain ufw-before-logging-forward (1 references)
target     prot opt source               destination         

Chain ufw-before-logging-input (1 references)
target     prot opt source               destination         

Chain ufw-before-logging-output (1 references)
target     prot opt source               destination         

Chain ufw-before-output (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ufw-user-output  all  --  anywhere             anywhere            

Chain ufw-logging-allow (0 references)
target     prot opt source               destination         
LOG        all  --  anywhere             anywhere             limit: avg 3/min burst 10 LOG level warning prefix "[UFW ALLOW] "

Chain ufw-logging-deny (2 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere             ctstate INVALID limit: avg 3/min burst 10
LOG        all  --  anywhere             anywhere             limit: avg 3/min burst 10 LOG level warning prefix "[UFW BLOCK] "

Chain ufw-not-local (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL
RETURN     all  --  anywhere             anywhere             ADDRTYPE match dst-type MULTICAST
RETURN     all  --  anywhere             anywhere             ADDRTYPE match dst-type BROADCAST
ufw-logging-deny  all  --  anywhere             anywhere             limit: avg 3/min burst 10
DROP       all  --  anywhere             anywhere            

Chain ufw-reject-forward (1 references)
target     prot opt source               destination         

Chain ufw-reject-input (1 references)
target     prot opt source               destination         

Chain ufw-reject-output (1 references)
target     prot opt source               destination         

Chain ufw-skip-to-policy-forward (0 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            

Chain ufw-skip-to-policy-input (7 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            

Chain ufw-skip-to-policy-output (0 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            

Chain ufw-track-forward (1 references)
target     prot opt source               destination         

Chain ufw-track-input (1 references)
target     prot opt source               destination         

Chain ufw-track-output (1 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             anywhere             ctstate NEW
ACCEPT     udp  --  anywhere             anywhere             ctstate NEW

Chain ufw-user-forward (1 references)
target     prot opt source               destination         

Chain ufw-user-input (1 references)
target     prot opt source               destination         
ACCEPT     udp  --  anywhere             anywhere             udp dpt:4789
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:7946
ACCEPT     udp  --  anywhere             anywhere             udp dpt:7946
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:2377
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh
ACCEPT     udp  --  anywhere             anywhere             udp dpt:22
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http
ACCEPT     udp  --  anywhere             anywhere             udp dpt:80
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https
ACCEPT     udp  --  anywhere             anywhere             udp dpt:443

Chain ufw-user-limit (0 references)
target     prot opt source               destination         
LOG        all  --  anywhere             anywhere             limit: avg 3/min burst 5 LOG level warning prefix "[UFW LIMIT BLOCK] "
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

Chain ufw-user-limit-accept (0 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            

Chain ufw-user-logging-forward (0 references)
target     prot opt source               destination         

Chain ufw-user-logging-input (0 references)
target     prot opt source               destination         

Chain ufw-user-logging-output (0 references)
target     prot opt source               destination         

Chain ufw-user-output (1 references)
target     prot opt source               destination 

Ports

ronin@ninjaserver:~$ sudo netstat -tulpen | grep LISTEN                                                                                                                               [sudo] password for ronin:                                                                                                                               tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      101        252122394  143/systemd-resolve                                                                                                                               tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      0          252124036  216/sshd: /usr/sbin                                                                                                                               tcp6       0      0 :::22                   :::*                    LISTEN      0          252124038  216/sshd: /usr/sbin                                                                                                                               tcp6       0      0 :::443                  :::*                    LISTEN      0          253418182  214/dockerd                                                                                                                               tcp6       0      0 :::9001                 :::*                    LISTEN      0          252131667  1080/docker-proxy                                                                                                                               tcp6       0      0 :::2377                 :::*                    LISTEN      0          252122897  214/dockerd                                                                                                                               tcp6       0      0 :::7946                 :::*                    LISTEN      0          252125261  214/dockerd                                                                                                                               tcp6       0      0 :::80                   :::*                    LISTEN      0          253418155  214/dockerd

Letsdebug error logs for traefik subdomain

https://letsdebug.net/traefik.whiskeyonthe.rocks/353757?debug=y
https://letsdebug.net/traefik.whiskeyonthe.rocks/353758?debug=y
https://letsdebug.net/traefik.whiskeyonthe.rocks/353760?debug=y

Letsdebug error logs for main domain

https://letsdebug.net/whiskeyonthe.rocks/353755?debug=y
https://letsdebug.net/whiskeyonthe.rocks/353768?debug=y
https://letsdebug.net/whiskeyonthe.rocks/353769?debug=y

Error logs

ronin@ninjaserver:~$ docker service logs traefik_traefik -f --no-trunc | grep -i error
traefik_traefik.1.nhylqf8qb89v7znftyfhgxixz@ninjaserver    | time="2020-11-11T19:27:30Z" level=error msg="Unable to obtain ACME certificate for domains \"traefik.whiskeyonthe.rocks\": unable to generate a certificate for the domains [traefik.whiskeyonthe.rocks]: error: one or more domains had a problem:\n[traefik.whiskeyonthe.rocks] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Connection refused, url: \n" providerName=le.acme rule="Host(`traefik.whiskeyonthe.rocks`)" routerName=reverse-proxy-https@docker

tcptraceroute resuts

user@host:~$ tcptraceroute 185.193.127.125 80
Selected device eth0, address 10.137.0.10, port 59381 for outgoing packets
Tracing the path to 185.193.127.125 on TCP port 80 (http), 30 hops max
 1  185.193.127.125 [open]  0.550 ms  0.338 ms  0.383 ms
user@host:~$ tcptraceroute 185.193.127.125 443
Selected device eth0, address 10.137.0.10, port 41519 for outgoing packets
Tracing the path to 185.193.127.125 on TCP port 443 (https), 30 hops max
 1  185.193.127.125 [open]  0.567 ms  0.360 ms  0.326 ms
user@host:~$ tcptraceroute traefik.whiskeyonthe.rocks 80
Selected device eth0, address 10.137.0.10, port 52493 for outgoing packets
Tracing the path to traefik.whiskeyonthe.rocks (185.193.127.125) on TCP port 80 (http), 30 hops max
 1  185.193.127.125 [open]  0.542 ms  0.401 ms  0.384 ms
user@host:~$ tcptraceroute traefik.whiskeyonthe.rocks 443
Selected device eth0, address 10.137.0.10, port 48251 for outgoing packets
Tracing the path to traefik.whiskeyonthe.rocks (185.193.127.125) on TCP port 443 (https), 30 hops max
 1  185.193.127.125 [open]  0.502 ms  0.380 ms  0.310 ms

remote CURL-ing the ports give me these:

user@host:~$ curl -v telnet://185.192.127.125:80
* Expire in 0 ms for 6 (transfer 0x60e552cab5b0)
*   Trying 185.192.127.125...
* TCP_NODELAY set
* Immediate connect fail for 185.192.127.125: Connection refused
* Closing connection 0
curl: (7) Couldn't connect to server
user@host:~$ curl -v telnet://traefik.whiskeyonthe.rocks:80
* Expire in 200 ms for 1 (transfer 0x5bd5112b95b0)
*   Trying 185.193.127.125...
* TCP_NODELAY set
* Immediate connect fail for 185.193.127.125: Connection refused
* Closing connection 0
curl: (7) Couldn't connect to server

localhost CURL-ing ports:

ronin@ninjaserver:~$ curl -v telnet://127.0.0.1:80
*   Trying 127.0.0.1:80...
* TCP_NODELAY set
* connect to 127.0.0.1 port 80 failed: Connection refused
* Failed to connect to 127.0.0.1 port 80: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 127.0.0.1 port 80: Connection refused
ronin@ninjaserver:~$ curl -v telnet://127.0.0.1:443
*   Trying 127.0.0.1:443...
* TCP_NODELAY set
* connect to 127.0.0.1 port 443 failed: Connection refused
* Failed to connect to 127.0.0.1 port 443: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 127.0.0.1 port 443: Connection refused
ronin@ninjaserver:~$ curl -v telnet://127.0.0.1:8080
*   Trying 127.0.0.1:8080...
* TCP_NODELAY set
* connect to 127.0.0.1 port 8080 failed: Connection refused
* Failed to connect to 127.0.0.1 port 8080: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 127.0.0.1 port 8080: Connection refused

It seems to me that the ports are in fact open but there is no active server listening on the other end of the ports? Which doesn't make sense to me since I am actively trying to deploy traefik docker service and I only have portainer agent service running on my docker swarm and I ensured that no other service would need ports 443 and 80. I am very confused at this point and I have no clue what's the best way for me to move forward. I would appreciate any assistance regarding this matter.

I don't think docker has bound the ports.

Try quoting the ports in the compose-file.

sudo ss -tnlp or sudo netstat -tnlp will show listening ports and processes, though by your other diagnosis you know this already.

Here is the inspection of my docker container after the change

ronin@ninjaserver:~/docker-traefik-letsencrypt$ docker service inspect traefik_traefik
[
    {
        "ID": "cjm16392lxu45jfu5zddb8gd6",
        "Version": {
            "Index": 65499
        },
        "CreatedAt": "2020-11-11T21:33:20.982361515Z",
        "UpdatedAt": "2020-11-11T21:33:22.184796102Z",
        "Spec": {
            "Name": "traefik_traefik",
            "Labels": {
                "com.docker.stack.image": "traefik:2.3.2",
                "com.docker.stack.namespace": "traefik",
                "constraint-label": "reverse-proxy",
                "traefik.docker.network": "reverse-proxy",
                "traefik.enable": "true",
                "traefik.http.middlewares.admin-auth.basicauth.users": "admin:$apr1$U1rooPiN$2uZapgnnZh./6ZBB9WTYq/",
                "traefik.http.routers.reverse-proxy-https.entrypoints": "web,web-secure",
                "traefik.http.routers.reverse-proxy-https.middlewares": "admin-auth",
                "traefik.http.routers.reverse-proxy-https.rule": "Host(`traefik.whiskeyonthe.rocks`)",
                "traefik.http.routers.reverse-proxy-https.service": "api@internal",
                "traefik.http.routers.reverse-proxy-https.tls": "true",
                "traefik.http.routers.reverse-proxy-https.tls.certresolver": "le",
                "traefik.http.routers.reverse-proxy.tls.domains[0].main": "whiskeyonthe.rocks",
                "traefik.http.routers.reverse-proxy.tls.domains[0].sans": "*.whiskeyonthe.rocks",
                "traefik.http.services.reverse-proxy.loadbalancer.server.port": "8080"
            },
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "traefik:2.3.2@sha256:6e6d4dc5a19afe06778ca092cdbbb98e31cb9f9c313edafa23f81a0e6ddf8a23",
                    "Labels": {
                        "com.docker.stack.namespace": "traefik"
                    },
                    "Args": [
                        "--accesslog",
                        "--log",
                        "--log.level=DEBUG",
                        "--api",
                        "--providers.docker",   
                        "--providers.docker.constraints=Label(`constraint-label`, `reverse-proxy`)",
                        "--providers.docker.exposedbydefault=false",
                        "--providers.docker.swarmmode",
                        "--entrypoints.web.address=:80",
                        "--entrypoints.web.http.redirections.entryPoint.to=web-secure",
                        "--entrypoints.web.http.redirections.entrypoint.scheme=https",
                        "--entrypoints.web-secure.address=:443",
                        "--certificatesresolvers.le.acme.email=admin@whiskeyonthe.rocks",
                        "--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json",
                        "--certificatesresolvers.le.acme.tlschallenge=true"
                    ],
                    "Env": [
                        "DOMAIN=traefik.whiskeyonthe.rocks",
                        "EMAIL=admin@whiskeyonthe.rocks",
                        "EXTRA_HOSTS=\"traefik.whiskeyonthe.rocks:185.193.127.125\"",
                        "HASHED_PASSWORD='$apr1$6XaPhlNf$N.NgCREOAXsVNmV8IKk//w.'",
                        "NODE_ID=63glxibxvxp4wvqpyrvnndvjt",
                        "PASSWORD=\"changethis\"",
                        "SRV_VPATH=\"/srv/traefik.whiskeyonthe.rocks/\"",
                        "USERNAME=admin"
                    ],
                    "Privileges": {
                        "CredentialSpec": null, 
                        "SELinuxContext": null  
                    },
                    "Mounts": [
                        {
                            "Type": "bind",
                            "Source": "/var/run/docker.sock",
                            "Target": "/var/run/docker.sock",
                            "ReadOnly": true
                        },
                        {
                            "Type": "bind",
                            "Source": "/srv/traefik.whiskeyonthe.rocks/certs",
                            "Target": "/certificates"
                        },
                        {
                            "Type": "bind",
                            "Source": "/srv/traefik.whiskeyonthe.rocks/letsencrypt",
                            "Target": "/letsencrypt"
                        }
                    ],
                    "StopGracePeriod": 10000000000,
                    "DNSConfig": {},
                    "Isolation": "default"
                },
                "Resources": {
                    "Limits": {
                        "NanoCPUs": 500000000, 
                        "MemoryBytes": 536870912
                    },
                    "Reservations": {
                        "NanoCPUs": 250000000,  
                        "MemoryBytes": 268435456
                    }
                },
                "RestartPolicy": {
                    "Condition": "any",
                    "Delay": 5000000000,
                    "MaxAttempts": 0
                },
                "Placement": {
                    "Constraints": [
                        "node.labels.reverse-proxy.reverse-proxy-certificates == true",
                        "node.role == manager"  
                    ],
                    "Platforms": [
                        {
                            "Architecture": "amd64",
                            "OS": "linux"
                        },
                        {
                            "OS": "linux"
                        },
                        {
                            "Architecture": "arm64",
                            "OS": "linux"
                        }
                    ]
                },
                "Networks": [
                    {
                        "Target": "fqtnk83r8sgi2toz0vb6macvz",
                        "Aliases": [
                            "traefik"
                        ]
                    }
                ],
                "ForceUpdate": 0,
                "Runtime": "container"
            },
            "Mode": {
                "Replicated": {
                    "Replicas": 1
                }
            },
            "UpdateConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "RollbackConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "EndpointSpec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "tcp",
                        "TargetPort": 80,
                        "PublishedPort": 80,
                        "PublishMode": "ingress"
                    },
                    {
                        "Protocol": "tcp",
                        "TargetPort": 443,
                        "PublishedPort": 443,
                        "PublishMode": "ingress"
                    }
                ]
            }
        },
        "Endpoint": {
            "Spec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "tcp",
                        "TargetPort": 80,
                        "PublishedPort": 80,
                        "PublishMode": "ingress"
                    },
                    {
                        "Protocol": "tcp",
                        "TargetPort": 443,
                        "PublishedPort": 443,
                        "PublishMode": "ingress"
                    }
                ]
            },
            "Ports": [
                {
                    "Protocol": "tcp",
                    "TargetPort": 80,
                    "PublishedPort": 80,
                    "PublishMode": "ingress"
                },
                {
                    "Protocol": "tcp",
                    "TargetPort": 443,
                    "PublishedPort": 443,
                    "PublishMode": "ingress"
                }
            ],
            "VirtualIPs": [
                {
                    "NetworkID": "pmeaxo2xvkulmkmjc4l2ow0z8",
                    "Addr": "10.0.0.12/24"
                },
                {
                    "NetworkID": "fqtnk83r8sgi2toz0vb6macvz",
                    "Addr": "10.0.2.31/24"
                }
            ]
        }
    }
]

docker-compose.yml

ronin@ninjaserver:~/docker-traefik-letsencrypt$ cat docker-compose.yml
version: '3.7'

services:

  traefik:
    image: traefik:2.3.2
    ports:
      - "80:80"
      - "443:443"
    deploy:
      resources:
        limits:
          cpus: '0.50'
          memory: 512M
        reservations:
          cpus: '0.25'
          memory: 256M
      placement:
        constraints:
          # Make the traefik service run only on the node with this label
          # as the node with it has the volume for the certificates
          - node.labels.reverse-proxy.reverse-proxy-certificates == true
          - node.role == manager
      labels:
        - constraint-label=reverse-proxy

        - traefik.enable=true
        - traefik.docker.network=reverse-proxy  

        - traefik.http.middlewares.admin-auth.basicauth.users=${USERNAME}:${HASHED_PASSWORD}

        - traefik.http.routers.reverse-proxy-https.rule=Host(`${DOMAIN}`)
        - traefik.http.routers.reverse-proxy-https.entrypoints=web,web-secure
        - traefik.http.routers.reverse-proxy-https.tls=true
        - traefik.http.routers.reverse-proxy-https.service=api@internal
        - traefik.http.routers.reverse-proxy-https.tls.certresolver=le
        - traefik.http.routers.reverse-proxy-https.middlewares=admin-auth
        - traefik.http.routers.reverse-proxy.tls.domains[0].main=whiskeyonthe.rocks
        - traefik.http.routers.reverse-proxy.tls.domains[0].sans=*.whiskeyonthe.rocks

        - traefik.http.services.reverse-proxy.loadbalancer.server.port=8080

    volumes:
      # Add Docker as a mounted volume, so that Traefik can read the labels of other services
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      # Mount the volume to store the certificates
      - "/srv/traefik.whiskeyonthe.rocks/certs:/certificates"
      - "/srv/traefik.whiskeyonthe.rocks/letsencrypt:/letsencrypt"
      #- "reverse-proxy-certificates:/certificates"
      #- "reverse-proxy-certificates:/letsencrypt"
    command:
      - --accesslog

      - --log
      - --log.level=DEBUG

      - --api

      - --providers.docker
      - --providers.docker.constraints=Label(`constraint-label`, `reverse-proxy`)
      - --providers.docker.exposedbydefault=false
      - --providers.docker.swarmmode

      - --entrypoints.web.address=:80
      # global redirect to https
      - --entrypoints.web.http.redirections.entryPoint.to=web-secure
      - --entrypoints.web.http.redirections.entrypoint.scheme=https
      - --entrypoints.web-secure.address=:443

      - --certificatesresolvers.le.acme.email=${EMAIL}
      - --certificatesresolvers.le.acme.storage=/letsencrypt/acme.json
      - --certificatesresolvers.le.acme.tlschallenge=true
      #- --certificatesresolvers.le.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
    networks:
      # Use the public network created to be shared between Traefik and
      # any other service that needs to be publicly available with HTTPS
      - reverse-proxy
    extra_hosts:
      - EXTRA_HOSTS
    environment:
      - NODE_ID=54fsdfdsfsdfsdfsfsdf
      - EMAIL=admin@whiskeyonthe.rocks

      - DOMAIN=traefik.whiskeyonthe.rocks
      - USERNAME=admin
      - PASSWORD="changethis"
      - HASHED_PASSWORD='$$apr1$$6XaPhlNf$$N.NgCREOAXsVNmV8IKk//w.'
      #- HASHPASS_ENV: $(HASHPASS_HOST:~(openssl passwd -apr1 $PASSWORD))
      #- $SRV_VPATH="/srv/DOMAIN.TLD/"
      - EXTRA_HOSTS="traefik.whiskeyonthe.rocks:185.193.127.125"
    env_file:
     - ./.env

  whoami:
    image: "traefik/whoami"
    container_name: "simple-service"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`whoami.whiskeyonthe.rocks`)"
      - "traefik.http.routers.whoami.middlewares=auth"
      - "traefik.http.routers.whoami.entrypoints=web,web-secure"
      - "traefik.http.routers.whoami.tls.certresolver=le"
    networks:
      - reverse-proxy

volumes:
  # Create a volume to store the certificates, there is a constraint to make sure
  # Traefik is always deployed to the same Docker node with the same volume containing
  # the HTTPS certificates
  reverse-proxy-certificates:
networks:
  # Use the previously created public network "reverse-proxy", shared with other
  # services that need to be publicly available via this Traefik
  reverse-proxy:
    external: true
    driver: overlay

Error logs

ronin@ninjaserver:~$ docker service logs traefik_traefik -f --no-trunc | grep -i error
traefik_traefik.1.87i0qkhi00evxli7mc0bnrq5r@ninjaserver    | time="2020-11-11T22:29:43Z" level=error msg="Unable to obtain ACME certificate for domains \"traefik.whiskeyonthe.rocks\": unable to generate a certificate for the domains [traefik.whiskeyonthe.rocks]: error: one or more domains had a problem:\n[traefik.whiskeyonthe.rocks] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Connection refused, url: \n" providerName=le.acme routerName=reverse-proxy-https@docker rule="Host(`traefik.whiskeyonthe.rocks`)"

ports

ronin@ninjaserver:~/docker-traefik-letsencrypt$ sudo ss -tnlp
[sudo] password for ronin: 
State                     Recv-Q                    Send-Q                                       Local Address:Port                                        Peer Address:Port                    Process                                                       
LISTEN                    0                         128                                          127.0.0.53%lo:53                                               0.0.0.0:*                        users:(("systemd-resolve",pid=143,fd=13))                    
LISTEN                    0                         128                                                0.0.0.0:22                                               0.0.0.0:*                        users:(("sshd",pid=216,fd=3))                                
LISTEN                    0                         128                                                   [::]:22                                                  [::]:*                        users:(("sshd",pid=216,fd=4))                                
LISTEN                    0                         128                                                      *:443                                                    *:*                        users:(("dockerd",pid=214,fd=55))                            
LISTEN                    0                         128                                                      *:9001                                                   *:*                        users:(("docker-proxy",pid=1080,fd=4))                       
LISTEN                    0                         128                                                      *:2377                                                   *:*                        users:(("dockerd",pid=214,fd=16))                            
LISTEN                    0                         128                                                      *:7946                                                   *:*                        users:(("dockerd",pid=214,fd=29))                            
LISTEN                    0                         128                                                      *:80                                                     *:*                        users:(("dockerd",pid=214,fd=54)) 
ronin@ninjaserver:~/docker-traefik-letsencrypt$ sudo netstat -tnlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      143/systemd-resolve 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      216/sshd: /usr/sbin 
tcp6       0      0 :::22                   :::*                    LISTEN      216/sshd: /usr/sbin 
tcp6       0      0 :::443                  :::*                    LISTEN      214/dockerd         
tcp6       0      0 :::9001                 :::*                    LISTEN      1080/docker-proxy   
tcp6       0      0 :::2377                 :::*                    LISTEN      214/dockerd         
tcp6       0      0 :::7946                 :::*                    LISTEN      214/dockerd         
tcp6       0      0 :::80                   :::*                    LISTEN      214/dockerd  

Look okay for the port binding, can you connect on localhost now?

Still looks like an internet connecting to your docker issue.

Localhost seems to give me the same issue.

I agree with it being an internet connecting to my docker issue. For some reason there doesn't seem to be any service listening and serving the ports via the reverse-proxy network from the docker ingress if I am interpreting and understanding correctly.

ronin@ninjaserver:~$ curl localhost:80
curl: (7) Failed to connect to localhost port 80: Connection refused
ronin@ninjaserver:~$ curl localhost:443
curl: (7) Failed to connect to localhost port 443: Connection refused

traefik_traefik service inspection

ronin@ninjaserver:~$ docker service inspect --pretty traefik_traefik

ID:             s6nr6hmvr4ymtm67vr59kp5ol
Name:           traefik_traefik
Labels:
 com.docker.stack.image=traefik:2.3.2
 com.docker.stack.namespace=traefik
 constraint-label=reverse-proxy
 traefik.docker.network=reverse-proxy
 traefik.enable=true
 traefik.http.middlewares.admin-auth.basicauth.users=admin:$apr1$U1rooPiN$2uZapgnnZh./6ZBB9WTYq/
 traefik.http.routers.reverse-proxy-https.entrypoints=web,web-secure
 traefik.http.routers.reverse-proxy-https.middlewares=admin-auth
 traefik.http.routers.reverse-proxy-https.rule=Host(`traefik.whiskeyonthe.rocks`)
 traefik.http.routers.reverse-proxy-https.service=api@internal
 traefik.http.routers.reverse-proxy-https.tls=true
 traefik.http.routers.reverse-proxy-https.tls.certresolver=le
 traefik.http.routers.reverse-proxy.tls.domains[0].main=whiskeyonthe.rocks
 traefik.http.routers.reverse-proxy.tls.domains[0].sans=*.whiskeyonthe.rocks
 traefik.http.services.reverse-proxy.loadbalancer.server.port=8080
Service Mode:   Replicated
 Replicas:      1
Placement:
 Constraints:   [node.labels.reverse-proxy.reverse-proxy-certificates == true node.role == manager]
UpdateConfig:
 Parallelism:   1
 On failure:    pause
 Monitoring Period: 5s
 Max failure ratio: 0
 Update order:      stop-first
RollbackConfig:
 Parallelism:   1
 On failure:    pause
 Monitoring Period: 5s
 Max failure ratio: 0
 Rollback order:    stop-first
ContainerSpec:
 Image:         traefik:2.3.2@sha256:6e6d4dc5a19afe06778ca092cdbbb98e31cb9f9c313edafa23f81a0e6ddf8a23
 Args:          --accesslog --log --log.level=DEBUG --api --providers.docker --providers.docker.constraints=Label(`constraint-label`, `reverse-proxy`) --providers.docker.exposedbydefault=false --providers.docker.swarmmode --entrypoints.web.address=:80 --entrypoints.web.http.redirections.entryPoint.to=web-secure --entrypoints.web.http.redirections.entrypoint.scheme=https --entrypoints.web-secure.address=:443 --certificatesresolvers.le.acme.email=admin@whiskeyonthe.rocks --certificatesresolvers.le.acme.storage=/letsencrypt/acme.json --certificatesresolvers.le.acme.tlschallenge=true 
 Env:           DOMAIN=traefik.whiskeyonthe.rocks EMAIL=admin@whiskeyonthe.rocks EXTRA_HOSTS="traefik.whiskeyonthe.rocks:185.193.127.125" HASHED_PASSWORD='$apr1$6XaPhlNf$N.NgCREOAXsVNmV8IKk//w.' NODE_ID=53glxibxvxp5wvqpyrvnndvjt PASSWORD="changethis" SRV_VPATH="/srv/traefik.whiskeyonthe.rocks/" USERNAME=admin 
Mounts:
 Target:        /var/run/docker.sock
  Source:       /var/run/docker.sock
  ReadOnly:     true
  Type:         bind
 Target:        /certificates
  Source:       /srv/traefik.whiskeyonthe.rocks/certs
  ReadOnly:     false
  Type:         bind
 Target:        /letsencrypt
  Source:       /srv/traefik.whiskeyonthe.rocks/letsencrypt
  ReadOnly:     false
  Type:         bind
Resources:
 Reservations:
  CPU:          0.25
  Memory:       256MiB
 Limits:
  CPU:          0.5
  Memory:       512MiB
Networks: reverse-proxy 
Endpoint Mode:  vip
Ports:
 PublishedPort = 80
  Protocol = tcp
  TargetPort = 80
  PublishMode = ingress
 PublishedPort = 443
  Protocol = tcp
  TargetPort = 443
  PublishMode = ingress 

docker network reverse-proxy inspection

ronin@ninjaserver:~$ docker network inspect reverse-proxy
[
    {
        "Name": "reverse-proxy",
        "Id": "o65nkhxhb3nope80z59h62vjt",
        "Created": "2020-11-11T22:41:52.278715403Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.2.0/24",
                    "Gateway": "10.0.2.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "13ad758a456d80bc193b60b574a44437d1a6df17090607cc85851fd4a93c56f1": {
                "Name": "traefik_traefik.1.21jiznw7hw6c2k41gjqllsbqh",
                "EndpointID": "3e15ad7eafcb811752c18f0005cc6b17740927003e67dab65da3a2a728bf390e",
                "MacAddress": "02:42:0a:00:02:03",
                "IPv4Address": "10.0.2.3/24",
                "IPv6Address": ""
            },
            "lb-reverse-proxy": {
                "Name": "reverse-proxy-endpoint",
                "EndpointID": "9edcd345aafe3b7855891673e03af9420a74c0d3c90b4ed34ac0cce9c8aa5f3f",
                "MacAddress": "02:42:0a:00:02:06",
                "IPv4Address": "10.0.2.6/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4098"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "3ffbee930b8f",
                "IP": "185.193.127.125"
            },
            {
                "Name": "dbec816330ec",
                "IP": "185.193.127.141"
            }
        ]
    }
]

netcat results

ronin@ninjaserver:~$ nc -vz 185.193.127.125 80
nc: connect to 185.193.127.125 port 80 (tcp) failed: Connection refused
ronin@ninjaserver:~$ nc -vz 185.193.127.125 443
nc: connect to 185.193.127.125 port 443 (tcp) failed: Connection refused

Its definitely a network issue with the swarm network. I removed a node I had from the docker swarm and now I am able to netcat, tcptraceroute and nmap the ports to the removed node without any issues since its no longer a node on the swarm.

Any idea how I can investigate this further?
It seems to be a common swarm issue as seen here:

But none of the solutions employed there are working for me right now. I have already defined a network in the docker-compose.yml and I have ensured that I used quotes on the ports "80:80" and made sure the service is exposing and publishing the ports.

docker service ls

ronin@ninjaserver:~$ docker service ls 
ID                  NAME                    MODE                REPLICAS            IMAGE                    PORTS
iwh2qbl4lyb2        portainer-agent_agent   global              1/1                 portainer/agent:latest   
s6nr6hmvr4ym        traefik_traefik         replicated          1/1                 traefik:2.3.2            *:80->80/tcp, *:443->443/tcp
2de6gnfhjziq        traefik_whoami          replicated          1/1                 traefik/whoami:latest

Can't say I've run into it specifically. What linux are your running on?

I suggest a test with your firewall relaxed or off and the same with SELinux/AppArmor/Whatever.

Getting into docker + swarm vs actual traefik forum territory here .

1 Like
ronin@ninjaserver:~/docker-traefik-letsencrypt$ cat /etc/os-release                                                                   valid_lft forever preferred_lft forever
NAME="Ubuntu"                                                                                                                      inet6 2a0a:3840:1337:127:0:b9c1:7f7d:1337/64 scope global
VERSION="20.04.1 LTS (Focal Fossa)"                                                                                                   valid_lft forever preferred_lft forever
ID=ubuntu                                                                                                                          inet6 fe80::1037:b9ff:fec1:7f7d/64 scope link
ID_LIKE=debian                                                                                                                        valid_lft forever preferred_lft forever
PRETTY_NAME="Ubuntu 20.04.1 LTS"                                                                                               ronin@ninjaserver:/srv$ export PASSWORD=changethis
VERSION_ID="20.04"                                                                                                             ronin@ninjaserver:/srv$ openssl passwd -apr1 $PASSWORD
HOME_URL="https://www.ubuntu.com/"                                                                                             $apr1$.XpDUQQw$m6lq0kTXbUSfhuJi3iOPw1
SUPPORT_URL="https://help.ubuntu.com/"                                                                                         ronin@ninjaserver:/srv$ export PASSWORD=
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"                                                                            ronin@ninjaserver:/srv$ openssl passwd -apr1 $PASSWORD
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"                                            Password:
VERSION_CODENAME=focal                                                                                                         Verifying - Password:
UBUNTU_CODENAME=focal 

I will try that and see where it goes from there thanks for your help :slight_smile:

I think the issue I had here was, I had a matrix server running on one of my other manager nodes ( 2 manager nodes in swarm). Once I removed the other node from the docker swarm, now I am able to run this configuration (somewhat with few other non related connection timed out issues) via the dashboard API. If there are other services occupying the same ports in swarm, then docker swarm cannot create service without throwing errors like above.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.