Server gave HTTP response to HTTPS client (Traefik 2.5)

[moofik]

I get error in Google Chrome browser that my connection is not private when I trying to address my domain. Also I have the following error in the traefik container logs (there are the only logs I have related to traefik container, there are no other logs):

traefik    | time="2022-04-27T09:15:29Z" level=info msg="Configuration loaded from file: /etc/traefik/traefik.yml"
traefik    | time="2022-04-27T09:15:37Z" level=error msg="Unable to obtain ACME certificate for domains \"spl-bck.ru\": cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get \"https://acme-v02.api.letsencrypt.org/directory\": http: server gave HTTP response to HTTPS client" providerName=letsencrypt.acme routerName=traefik-router@docker rule="Host(`spl-bck.ru`)"

I use the following configuration for my docker-compose.yml on the server:

version: '3'
services:
  traefik:
    image: traefik:v2.5
    container_name: traefik
    restart: unless-stopped
    command: --go.insecure=true --providers.docker
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./traefik/traefik.yml:/etc/traefik/traefik.yml:ro
      - ./traefik/acme.json:/app/acme.json
    networks:
      - crypto

  web:
    image: moofik/chatbot_api:latest
    restart: always
    container_name: web
    environment:
      DB_HOST: example
      DB_USERNAME: example
      DB_PASSWORD: example
      DB_DATABASE: example
    ports:
      - "8181:8181"
    volumes:
      - ./:/var/run/docker.sock
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik-router.entrypoints=websecure"
      - "traefik.http.routers.traefik-router.rule=Host(`spl-bck.ru`)"
      - "traefik.http.routers.traefik-router.tls.certresolver=letsencrypt"
    networks:
      - crypto
    depends_on:
      - db
      - traefik

  db:
    build: postgres
    image: crypto/postgresql
    container_name: db
    environment:
      POSTGRES_PASSWORD: example
    volumes:
      - "./volumes/pgdata:/var/lib/postgresql/data"
      - "./volumes/data:/opt/data"
    tmpfs:
      - /run
      - /tmp
    ports:
      - "5432:5432"
    networks:
      - crypto

networks:
  crypto:
    external: true

Also, here is my traefik.yml

entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
          permanent: true
  websecure:
    address: ":443"

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false

certificatesResolvers:
  letsencrypt:
    acme:
      email: moofik12@gmail.com
      storage: /app/acme.json
      httpChallenge:
        entryPoint: web

What am I doing wrong and how can I fix my errors? Please, help.

[cakiwi]

What happens if you curl https://acme-v02.api.letsencrypt.org/directory from that server ?

[moofik]

I got the following error:
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number
Seems like alpine can't handle tls by some reason. Maybe there is wrong configuration for SSL on my server? What should I do?

[cakiwi]

@moofik

Seems like alpine can't handle tls. What should I do?

It really does.

Alpine cURL

docker run --rm -it alpine 
/ # apk -u add curl
fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/community/x86_64/APKINDEX.tar.gz
(1/11) Upgrading busybox (1.34.1-r3 -> 1.34.1-r5)
Executing busybox-1.34.1-r5.post-upgrade
(2/11) Upgrading libcrypto1.1 (1.1.1l-r7 -> 1.1.1n-r0)
(3/11) Upgrading libssl1.1 (1.1.1l-r7 -> 1.1.1n-r0)
(4/11) Upgrading libretls (3.3.4-r2 -> 3.3.4-r3)
(5/11) Upgrading ssl_client (1.34.1-r3 -> 1.34.1-r5)
(6/11) Upgrading zlib (1.2.11-r3 -> 1.2.12-r1)
(7/11) Installing ca-certificates (20211220-r0)
(8/11) Installing brotli-libs (1.0.9-r5)
(9/11) Installing nghttp2-libs (1.46.0-r0)
(10/11) Installing libcurl (7.80.0-r1)
(11/11) Installing curl (7.80.0-r1)
Executing busybox-1.34.1-r5.trigger
Executing ca-certificates-20211220-r0.trigger
OK: 8 MiB in 19 packages
/ # curl https://acme-v02.api.letsencrypt.org/directory
{
  "FcH__8mSzsk": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
}/ # 

Something is intercepting the tls connection between that server an LetsEncrypt. Common scenario would be a corporate proxy or a port redirect. The error received can often be received when a non-tls response is received during the TLS handshake.

e.g TLS connection to a standard HTTP port:

openssl s_client -connect google.com:80
CONNECTED(00000003)
140570879186240:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../ssl/record/ssl3_record.c:331:

[moofik]

Thank you, sir. You were right - I had the error because of my incorrect iptables rules.
But I have some misconception about traefik work. Could you please explain how should I pass all requests that coming to 443 port (Traefik container) to 8181 port (my web app) ? I thought I should do it with iptables somehow. Maybe I should do it with labels for docker? How can traefik know where should it pass the request?

[cakiwi]

No need to touch IP tables(beyond 443/80 for traefik. Some cases docker will open it anyway ).

Internet -> Traefik -> Backend Containers

Yes the labels are used to define the rules. As you're using the docker provider Traefik knows the backends ip (via docker api and the docker network).

I like their pictures