Request ECONNREFUSED 127.0.0.1:443

Hi,

I have a node web application with self-signed tls cert running on docker-compose and traefik. I am able to connect to the web application, but whenever I try to make a POST request from the browser I'm getting Error: connect ECONNREFUSED 127.0.0.1:443. I'm wondering what are some possible reasons for this error? From my understanding traefik should be expose on :443, and it seems to not be able to connect.

This is the output of the curl:

kakwong at LM-NYB-21014311 in ~/code/shabu on dev
$ curl -v -d 'username=kortina&password=tv' -kL dev.app.com/login
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to dev.app.com (127.0.0.1) port 80 (#0)
> POST /login HTTP/1.1
> Host: dev.app.com
> User-Agent: curl/7.54.0
> Accept: */*
> Content-Length: 28
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 28 out of 28 bytes
< HTTP/1.1 307 Temporary Redirect
< Location: https://dev.app.com/login
< Date: Thu, 24 Oct 2019 16:50:18 GMT
< Content-Length: 18
< Content-Type: text/plain; charset=utf-8
<
* Ignoring the response-body
* Connection #0 to host dev.app.com left intact
* Issue another request to this URL: 'https://dev.app.com/login'
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to dev.app.com (127.0.0.1) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=TRAEFIK DEFAULT CERT
*  start date: Oct 24 16:23:31 2019 GMT
*  expire date: Oct 23 16:23:31 2020 GMT
*  issuer: CN=TRAEFIK DEFAULT CERT
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fbd3f803c00)
> POST /login HTTP/2
> Host: dev.app.com
> User-Agent: curl/7.54.0
> Accept: */*
> Content-Length: 28
> Content-Type: application/x-www-form-urlencoded
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
* We are completely uploaded and fine
< HTTP/2 500
< access-control-allow-origin: https://dev.app.com
< content-type: application/json; charset=utf-8
< date: Thu, 24 Oct 2019 16:50:18 GMT
< etag: W/"22-vuNxeJ3uUTtL9oPjN9SXLohrZi8"
< set-cookie: v_id=fp01-6ce67451-9edb-4ef7-9999-80e3e790294e; Max-Age=157680000; Domain=dev.app.com; Path=/; Expires=Tue, 22 Oct 2024 16:50:18 GMT
< vary: X-HTTP-Method-Override, Origin
< x-powered-by: Express
< content-length: 34
<
* Connection #1 to host dev.app.com left intact
{"error":"There was an an error."}

Thanks!

Hi @kwngo, could you share the configuration (docker-compose definition of traefik and the node app, traefik's logs with debug enabled, any other useful information)?

The curl log you are providing is not mentioning the error ECONNREFUSED, so it is hard to understand where the problem is.

Hi @dduportal, thanks for the help!

docker-compose.yml

version: '3.2'

#TODO project name / prefix is dynamic
# only overrides are flag + env var, not this file

services:
  reverse-proxy:
    image: traefik:v2.0
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./traefik.yaml:/traefik.yaml
    labels:
      - "traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)"
      - "traefik.http.routers.http-catchall.entrypoints=web"
      - "traefik.http.routers.http-catchall.middlewares=redirect-to-https@docker"
      - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
  node:
    build:
      context: ../node
      dockerfile: ./docker/runtime.Dockerfile
      args:
        GITHUB_SSH_KEY: ghe_key
    environment:
      - NODE_ENV=development
      - NEW_RELIC_APP_NAME=''
      - NEW_RELIC_ENABLED=false
      - NEW_RELIC_LICENSE_KEY=''
      - PORT=9001
      - SES_KEY=''
      - SES_SECRET=''
      - ENVIRONMENT=development
      - NODE_TLS_REJECT_UNAUTHORIZED=0
    expose:
      - 9001
    labels:
      - "traefik.http.routers.node.rule=Host(`dev.app.com`)"
      - "traefik.http.routers.node.tls=true"
      - "traefik.http.middlewares.node-replace-path.replacepathregex.regex=/w/(.*)"
      - "traefik.http.middlewares.node-replace-path.replacepathregex.replacement=/$$1"
      - "traefik.http.routers.node.middlewares=node-replace-path@docker"
      - "traefik.http.middlewares.node-errorpage.errors.status=306"
      - "raefik.http.middlewares.node-errorpage.errors.service=web"
      - "traefik.http.services.node.loadbalancer.server.port=9001"
    command: npm start

node logs for POST /login

node_1          | 2019-10-25T15:37:20.215Z - error:  Error: connect ECONNREFUSED 127.0.0.1:443
node_1          |     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1106:14)
node_1          | 2019-10-25T15:37:20.220Z - request: ip="172.20.0.1" latency="81.823" method="POST" path="/login" request="d8a5d0b2-52f5-4e27-b60c-9710d3bff4d3" authenticated="false" service="node" size="34" status="500" type="server" url="/login" referrer="https://dev.app.com/" browser="Chrome 77.0.3865.120" os="Mac OS 10.14.6" exception="connect ECONNREFUSED 127.0.0.1:443"

Treafik debug logs

time="2019-10-25T15:37:20Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/login\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"application/json\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,en;q=0.9\"],\"Content-Length\":[\"55\"],\"Content-Type\":[\"application/x-www-form-urlencoded\"],\"Cookie\":[\"v_id=fp01-58e05dd6-3048-4714-814f-6f7c8dbde0d1; _ga=GA1.2.1292558718.1571938967; _gid=GA1.2.696183587.1571938967; _gat_gtag_UA_15492939_14=1; _gat_gtag_UA_15492939_15=1\"],\"Origin\":[\"https://dev.app.com\"],\"Referer\":[\"https://dev.app.com/\"],\"Sec-Fetch-Mode\":[\"cors\"],\"Sec-Fetch-Site\":[\"same-origin\"],\"User-Agent\":[\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36\"],\"X-Forwarded-Host\":[\"dev.app.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"6ccde9b91817\"],\"X-Real-Ip\":[\"172.20.0.1\"]},\"ContentLength\":55,\"TransferEncoding\":null,\"Host\":\"dev.app.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.20.0.1:39328\",\"RequestURI\":\"/login\",\"TLS\":null}"
time="2019-10-25T15:37:20Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/login\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"application/json\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,en;q=0.9\"],\"Content-Length\":[\"55\"],\"Content-Type\":[\"application/x-www-form-urlencoded\"],\"Cookie\":[\"v_id=fp01-58e05dd6-3048-4714-814f-6f7c8dbde0d1; _ga=GA1.2.1292558718.1571938967; _gid=GA1.2.696183587.1571938967; _gat_gtag_UA_15492939_14=1; _gat_gtag_UA_15492939_15=1\"],\"Origin\":[\"https://dev.app.com\"],\"Referer\":[\"https://dev.app.com/\"],\"Sec-Fetch-Mode\":[\"cors\"],\"Sec-Fetch-Site\":[\"same-origin\"],\"User-Agent\":[\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36\"],\"X-Forwarded-Host\":[\"dev.app.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"6ccde9b91817\"],\"X-Real-Ip\":[\"172.20.0.1\"]},\"ContentLength\":55,\"TransferEncoding\":null,\"Host\":\"dev.app.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.20.0.1:39328\",\"RequestURI\":\"/login\",\"TLS\":null}" ForwardURL="http://172.20.0.10:9001"
time="2019-10-25T15:37:20Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/login\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"application/json\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,en;q=0.9\"],\"Content-Length\":[\"55\"],\"Content-Type\":[\"application/x-www-form-urlencoded\"],\"Cookie\":[\"v_id=fp01-58e05dd6-3048-4714-814f-6f7c8dbde0d1; _ga=GA1.2.1292558718.1571938967; _gid=GA1.2.696183587.1571938967; _gat_gtag_UA_15492939_14=1; _gat_gtag_UA_15492939_15=1\"],\"Origin\":[\"https://dev.app.com\"],\"Referer\":[\"https://dev.app.com/\"],\"Sec-Fetch-Mode\":[\"cors\"],\"Sec-Fetch-Site\":[\"same-origin\"],\"User-Agent\":[\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36\"],\"X-Forwarded-Host\":[\"dev.app.com\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"6ccde9b91817\"],\"X-Real-Ip\":[\"172.20.0.1\"]},\"ContentLength\":55,\"TransferEncoding\":null,\"Host\":\"dev.app.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.20.0.1:39328\",\"RequestURI\":\"/login\",\"TLS\":null}"

Hi @kwngo, to me, it looks like that this error only comes form your node application:

  • the curl output of your initial messages shows that Express answers an HTTP/500 error. It means that Traefik forwarded the POST and the 500 answer.
  • The docker-compose logs from your node application shows that the ECONNECT error happens inside the node process, which explains why an HTTP/500 is answered.

=> Why is your NodeJS application trying to connect to 443 when it receives the POST?
If for any reason you want node to be able to request traefik, then you must use either the docker DNS (so it connects to https://reverse-proxy:443 ) or maybe the Docker bridge IP (which is 172.17.0.1 by default but it might change).

Hi @dduportal, we've narrowed down the issue to the ssl certificate, meaning it's at least not entirely traefik related. Thanks for the help and future debugging tips! :grinning:

1 Like