Intermittent Connection Failure

swiver · August 13, 2022, 6:27am

I have been getting a strange behavior with Traefik immediately upon switching to an ACME DNS Wildcard Certificate. All of my applications load correctly and get their TLS certs, and several minutes later most of them lose connection. I get ERR_CONNECTION_REFUSED in a Chrome browser when trying to access them and the debug log does not show a connection attempt.

It's like Traefik is not serving my applications at all in that time, and it can be anywhere from a few seconds to a few hours before they become accessible again. The debug logs are completely normal showing only successful traffic.

at all times two of my applications work perfectly while all of the rest are down, the same two.

The most curious thing is cached pages. If I have one of the apps cached and keep it open it will communicate correctly through Traefik until I force refresh. This shows in the log as normal traffic.

Does anybody have any idea what may be happening? I am using ACME DNS, GoDaddy with the domains A records pointed toward my server. The CNAME record is set correctly and I can confirm this using dig _acme-challenge.myDomainOne.com

I have installed the latest docker and traefik versions attempting to troubleshoot this problem:

Rocky Linux 8.6 Kernel: 4.18.0-372.19.1.el8_6.x86_64
Docker Compose version v2.9.0
Docker version 20.10.17, build 100c701
Traefik v2.8.3

My docker-compose.yml

services:
  portainer:
    container_name: portainer
    image: portainer/portainer
    restart: unless-stopped
    ports:
      - 9000:9000
    environment:
      - PGID=1000
      - PUID=1000
    volumes:
      - /home/myUser/.portainer:/data
      - /var/run/docker.sock:/var/run/docker.sock
  fail2ban:
    container_name: fail2ban
    restart: always
    image: crazymax/fail2ban:latest
    environment: 
      - F2B_DB_PURGE_AGE=30d
      - F2B_LOG_TARGET=/f2b_logs/dropped.log
      - F2B_LOG_LEVEL=INFO
      - F2B_IPTABLES_CHAIN=INPUT
    volumes:
      - /home/myUser/.fail2ban:/data
      - /home/myUser/.reverse_proxy/traefik/logs:/traefik/logs:ro
      - /home/myUser/.fail2ban/logs:/f2b_logs/
      - /home/myUser/.fail2ban/scripts:/scripts
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
    network_mode: "host"
    privileged: true
    cap_add:
      - NET_ADMIN
      - NET_RAW
  reverse-proxy:
    image: traefik:latest
    container_name: traefik
    restart: unless-stopped
    networks:
      - rproxy
    security_opt:
      - no-new-privileges:true
    ports:
      - 80:80
      - 443:443
    environment:
      - ACME_DNS_API_BASE=https://auth.acme-dns.io
      - ACME_DNS_STORAGE_PATH=/acme/acme-dns_storage.json
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /home/myUser/.reverse_proxy/traefik/traefik.yml:/traefik.yml:ro
      - /home/myUser/.reverse_proxy/traefik/config:/config
      - /home/myUser/.reverse_proxy/traefik/logs:/logs
      - /home/myUser/.reverse_proxy/traefik/acme:/acme
      - /home/myUser/.reverse_proxy/traefik/certs:/certs
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=rproxy"
      - "traefik.http.routers.traefik-secure.entrypoints=websecure"
      - "traefik.http.routers.traefik-secure.rule=Host(`traefik.myDomainOne.co`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik_panel-auth@file"
      - "traefik.http.routers.traefik-secure.service=api@internal"
#########################################################
#                 myDomainOne Subdomains                                                                    #
#########################################################
  regexgenerator:
    image: noxone/regexgenerator
    container_name: regexgenerator
    restart: unless-stopped
    ports:
      - "8186:80"
    networks:
      - rproxy
    labels:
      - "traefik.enable=true"
      - "traefik.port=80"
      - "traefik.docker.network=rproxy"
      - "traefik.http.routers.regexgen-secure.entrypoints=websecure"
      - "traefik.http.routers.regexgen-secure.rule=Host(`regex.myDomainOne.co`)"
  gitea:
    image: gitea/gitea:1.16.9
    container_name: gitea
    environment:
      - USER_UID=1000
      - USER_GID=1000
    restart: always
    networks:
      - rproxy
      - gitea
    volumes:
      - /home/myUser/.gitea:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "3070:3070"
      - "2272:22"
    labels:
      - "traefik.enable=true"
      - "traefik.port=3070"
      - "traefik.docker.network=rproxy"
      - "traefik.http.services.gitea-secure.loadbalancer.server.port=3070"
      - "traefik.http.routers.gitea-secure.entrypoints=websecure"
      - "traefik.http.routers.gitea-secure.rule=Host(`git.myDomainOne.co`)"
  bibliogram:
    image: quay.io/pussthecatorg/bibliogram:latest
    container_name: bibliogram
    networks:
      - rproxy
    volumes:
      - /home/myUser/.bibliogram/config.js:/app/config.js
      - /home/myUser/.bibliogram/db:/app/db
    ports:
      - 10407:10407
    depends_on:
      - reverse-proxy
    environment:
      - UID=1000
      - GID=1000
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      - "traefik.port=10407"
      - "traefik.docker.network=rproxy"
      - "traefik.http.routers.bibliogram-secure.entrypoints=websecure"
      - "traefik.http.routers.bibliogram-secure.rule=Host(`ig.myDomainOne.co`)"
  rocketchat:
    image: registry.rocket.chat/rocketchat/rocket.chat:${RELEASE:-latest}
    container_name: rocketchat
    restart: on-failure
    environment:
      PGID: 1000
      PUID: 1000
      MONGO_URL: mongodb://rocketchat-db:27017/rocketchat?replicaSet=rs0&ssl=false
      MONGO_OPLOG_URL: mongodb://rocketchat-db:27017/local
      ROOT_URL: https://rc.myDomainOne.co
      PORT: 3000
      Accounts_UseDNSDomainCheck: 'false'
    depends_on:
      - rocketchat-db
    expose:
      - 3000
    ports:
      - target: 3000
        published: 3000
        protocol: tcp
    networks:
      - rproxy
      - rocketchat
    labels:
      - "traefik.enable=true"
      - "traefik.port=3000"
      - "traefik.docker.network=rproxy"
      - "traefik.http.routers.rocketchat-secure.entrypoints=websecure"
      - "traefik.http.routers.rocketchat-secure.rule=Host(`rc.myDomainOne.co`)"
  rocketchat-db:
    image: docker.io/bitnami/mongodb:${MONGODB_VERSION:-4.4}
    container_name: rocketchat-db
    restart: on-failure
    volumes:
      - /home/myUser/.rocketchat/db:/bitnami/mongodb
    environment:
      PGID: 1000
      PUID: 1000
      MONGODB_REPLICA_SET_MODE: primary
      MONGODB_REPLICA_SET_NAME: ${MONGODB_REPLICA_SET_NAME:-rs0}
      MONGODB_PORT_NUMBER: ${MONGODB_PORT_NUMBER:-27017}
      MONGODB_INITIAL_PRIMARY_HOST: ${MONGODB_INITIAL_PRIMARY_HOST:-mongodb}
      MONGODB_INITIAL_PRIMARY_PORT_NUMBER: ${MONGODB_INITIAL_PRIMARY_PORT_NUMBER:-27017}
      MONGODB_ADVERTISED_HOSTNAME: ${MONGODB_ADVERTISED_HOSTNAME:-mongodb}
      MONGODB_ENABLE_JOURNAL: ${MONGODB_ENABLE_JOURNAL:-true}
      ALLOW_EMPTY_PASSWORD: ${ALLOW_EMPTY_PASSWORD:-yes}
    networks:
      - rocketchat
  rocketchat-webdav:
    image: bytemark/webdav
    container_name: rocketchat-webdav
    restart: always
    networks:
      - rocketchat
    ports:
      - "3080:80"
    environment:
      AUTH_TYPE: Digest
      ANONYMOUS_METHOS: ALL
      USERNAME: rocketchat
      PASSWORD: password
    volumes:
      - /home/myUser/.rocketchat/files:/var/lib/dav
  proxitok:
    container_name: proxitok
    image: quay.io/pussthecatorg/proxitok:latest
    restart: unless-stopped
    networks:
      - rproxy
      - tiktok
    ports:
      - "8181:80"
    environment:
      - "LATTE_CACHE=/cache"
      - "API_CACHE=redis"
      - "REDIS_HOST=proxitok-redis"
      - "REDIS_PORT=6379"
      - "API_SIGNER_URL=http://proxitok-signer:8080/signature"
    volumes:
      - "proxitok-cache:/cache"
    depends_on:
      - proxitok-redis
      - proxitok-signer
    labels:
      - "traefik.enable=true"
      - "traefik.port=80"
      - "traefik.docker.network=rproxy"
      - "traefik.http.routers.tiktok-secure.entrypoints=websecure"
      - "traefik.http.routers.tiktok-secure.rule=Host(`tiktok.myDomainOne.co`)"
  proxitok-redis:
    container_name: proxitok-redis
    image: docker.io/redis:6-alpine
    restart: unless-stopped
    networks:
      - tiktok
    command: redis-server --save 60 1 --loglevel warning
  proxitok-signer:
    container_name: proxitok-signer
    image: ghcr.io/pablouser1/signtok:master
    networks:
      - tiktok
  nitter:
    image: zedeus/nitter:latest
    container_name: nitter
    ports:
    - "8182:8182"
    volumes:
    - type: bind
      source: /home/myUser/.nitter/nitter.conf
      target: /src/nitter.conf
    depends_on:
      - nitter-redis
    restart: unless-stopped
    networks:
      - rproxy
      - nitter
    labels:
      - "traefik.enable=true"
      - "traefik.port=8182"
      - "traefik.docker.network=rproxy"
      - "traefik.http.routers.twitter-secure.entrypoints=websecure"
      - "traefik.http.routers.twitter-secure.rule=Host(`tweet.myDomainOne.co`)"
  nitter-redis:
    image: redis:6-alpine
    container_name: nitter-redis
    command: redis-server --save 60 1 --loglevel warning
    volumes:
      - nitter-redis:/data
    restart: unless-stopped
    networks:
      - nitter
    healthcheck:
      test: redis-cli ping
      interval: 30s
      timeout: 5s
      retries: 2
  teddit:
    container_name: teddit
    image: teddit/teddit:latest
    environment:
      - DOMAIN=reddit.myDomainOne.co
      - USE_HELMET=true
      - USE_HELMET_HSTS=true
      - TRUST_PROXY=true
      - REDIS_HOST=teddit-redis
      - THEME=nord
    ports:
      - "127.0.0.1:8185:8080"
    networks:
      - rproxy
      - teddit
    healthcheck:
      test: ["CMD", "wget" ,"--no-verbose", "--tries=1", "--spider", "http://localhost:8080/about"]
      interval: 1m
      timeout: 3s
    depends_on:
      - teddit-redis
    labels:
      - "traefik.enable=true"
      - "traefik.port=8080"
      - "traefik.docker.network=rproxy"
      - "traefik.http.routers.reddit-secure.entrypoints=websecure"
      - "traefik.http.routers.reddit-secure.rule=Host(`reddit.myDomainOne.co`)"
  teddit-redis:
    container_name: teddit-redis
    image: redis:6.2.5-alpine
    command: redis-server
    environment:
      - REDIS_REPLICATION_MODE=master
    networks:
      - teddit
  searxng:
    container_name: searxng
    image: searxng/searxng:latest
    networks:
      - searxng
      - rproxy
    volumes:
      - /home/myUser/.searx:/etc/searxng:rw
    environment:
      - SEARXNG_BASE_URL=https://search.myDomainOne.co
    cap_drop:
      - ALL
    cap_add:
      - CHOWN
      - SETGID
      - SETUID
      - DAC_OVERRIDE
    logging:
      driver: "json-file"
      options:
        max-size: "1m"
        max-file: "1"
    labels:
      - "traefik.enable=true"
      - "traefik.port=8183"
      - "traefik.docker.network=rproxy"
      - "traefik.http.routers.searx-secure.entrypoints=websecure"
      - "traefik.http.routers.searx-secure.rule=Host(`search.myDomainOne.co`)"
  searx-redis:
    container_name: searx-redis
    image: "redis:alpine"
    command: redis-server --save "" --appendonly "no"
    networks:
      - searxng
    tmpfs:
      - /var/lib/redis
    cap_drop:
      - ALL
    cap_add:
      - SETGID
      - SETUID
      - DAC_OVERRIDE
#####################################
#                   myDomainOne.co DAMP                        #
#             Docker Apache MariaDB PHP                      #
#####################################
  myDomainOne_php:
    build: 
      context: '/home/myUser/.myDomainOne/php/'
      args:
        - "PHP_VERSION=7.3"
    networks:
      - myDomainOne
    volumes:
      - /home/myUser/.myDomainOne/html:/var/www/html
      - /home/myUser/.myDomainOne/logs:/logs
    container_name: myUser_php
    restart: always
  myDomainOne_apache:
    build:
      context: '/home/myUser/.myDomainOne/apache/'
      args:
        - "APACHE_VERSION=2.4"
    depends_on:
      - myUser_php
      - myUser_mariadb
    networks:
      - myDomainOne
      - rproxy
    ports:
      - 5580:80
    volumes:
      - /home/myUser/.myDomainOne/html:/var/www/html
      - /home/myUser/.myDomainOne/logs:/logs
    container_name: myDomainOne_apache
    restart: always
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=rproxy"
      - "traefik.port=80"
      - "traefik.http.routers.myDomainOneapache-secure.entrypoints=websecure"
      - "traefik.http.routers.myDomainOneapache-secure.rule=Host(`myDomainOne.co`) || Host(`www.myDomainOne.co`)"
  myDomainOne_mariadb:
    image: mariadb
    restart: always
    ports:
      - 3306:3306
    volumes:
      - /home/myUser/.myDomainOne/mariadb:/var/lib/mysql
    networks:
      - myDomainOne
    env_file: /home/myUser/.myDomainOne/myDomainOne.env
    container_name: myDomainOne_mariadb
  myDomainThreeca_wordpress:
    image: wordpress
    restart: always
    depends_on:
      - myDomainThree_mariadb
    networks:
      - myDomainThree
      - rproxy
    ports:
      - 5680:80
    env_file: /home/myUser/.myDomainThree/myDomainThree.env
    volumes:
      - /home/myUser/.myDomainThree/html:/var/www/html
    container_name: myDomainThree_wordpress
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=rproxy"
      - "traefik.port=5680"
      - "traefik.http.routers.myDomainThreeapache-secure.entrypoints=websecure"
      - "traefik.http.routers.myDomainThreeapache-secure.rule=Host(`myDomainThree.ca`)"
      - "traefik.http.routers.myDomainThreeapache-secure.tls.certResolver=le-dns_myDomainThree"
      - "traefik.http.routers.myDomainThreeapache-secure.tls.domains[0].main=myDomainThree.ca"
      - "traefik.http.routers.myDomainThreeapache-secure.tls.domains[0].sans=*.myDomainThree.ca"
      - "traefik.http.middlewares.https-redirect.headers.sslproxyheaders.X-Forwarded-Proto=https"
  myDomainThreeca_mariadb:
    image: mariadb
    restart: always
    ports:
      - 5606:3306
    volumes:
      - /home/myUser/.myDomainThree/mariadb:/var/lib/mysql
    networks:
      - myDomainThree_ca
    env_file: /home/myUser/.myDomainThree/myDomainThree.env
    container_name: myDomainThree_mariadb
volumes:
  proxitok-cache:
  nitter-redis:
networks:
  meet.jitsi:
  rproxy:
    name: rproxy
    external: true
  myDomainOne:
  myDomainThree:
  tiktok:
    name: tiktok
  nitter:
    name: nitter
  searxng:
    ipam:
      driver: default
  gitea:
    external: false
  rocketchat:
    name: rocketchat
    external: false
  teddit:
    name: rocketchat
    external: false

My traefik.yml

api:
  dashboard: true
entryPoints:
  web:
    address: :80
    http:
      redirections:
        entryPoint:
          to: websecure
  websecure:
    address: :443
    http:
      middlewares:
        - secureHeaders@file
        - nofloc@file
      tls:
        certResolver: le-dns_myDomainOne
        domains:
          - main: myDomainOne.co
            sans:
              - "*.myDomainOne.co"
log:
  level: DEBUG
  filePath: "/logs/debug.log"
accessLog:
  filePath: "/logs/access.log"
  bufferingSize: 100
  filters:
    statusCodes:
      - "200"
      - "300-302"
      - "400-499"
    retryAttempts: true
  fields:
    defaultMode: keep
    names:
      StartUTC: drop
providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
  file:
    filename: /config/traefik_dynamic.yml
certificatesResolvers:
  le-dns_myDomainOne:
    acme:
      caServer: https://acme-v02.api.letsencrypt.org/directory
      email: myUser@protonmail.com
      storage: /acme/acme-dns_myDomainOne.json
      keyType: EC384
      dnsChallenge:
        delayBeforeCheck: 120
        provider: acme-dns
  le-dns_myDomainTwo:
    acme:
      caServer: https://acme-v02.api.letsencrypt.org/directory
      email: myUser@protonmail.com
      storage: /acme/acme-dns_myDomainTwo.json
      keyType: EC384
      dnsChallenge:
        delayBeforeCheck: 120
        provider: acme-dns
  le-dns_myDomainThree:
    acme:
      caServer: https://acme-v02.api.letsencrypt.org/directory
      email: myUser@protonmail.com
      storage: /acme/acme-dns_myDomainThree.json
      keyType: EC384
      dnsChallenge:
        delayBeforeCheck: 120
        provider: acme-dns

My traefik_dynamic.yml

# Dynamic configuration
http:
  middlewares:
    nofloc:
      headers:
        customResponseHeaders:
          Permissions-Policy: "interest-cohort=()"
    secureHeaders:
      headers:
        sslRedirect: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 31536000   
    traefik_panel-auth:
      basicAuth:
        users:
          - "myUser:$xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    plex-auth:
      basicAuth:
        users:
          - "myUser:$xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    jellyfin-auth:
      basicAuth:
        users:
          - "myUser:$xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    bw-stripPrefix:
      stripPrefix:
        prefixes:
          - "/notifications/hub"
        forceSlash: false
    redirect-7dtd:
      redirectRegex:
        regex: "^https://7dtd.myDomainOne.co"
        replacement: "steam://connect/7d.myDomainOne.co:26900"
    7dtd-troubleshoot:
      replacePath:
        path: "/7dtd"
    redirect-discord:
      redirectRegex:
        regex: "^https://discord.myDomainOne.co"
        replacement: "https://discord.gg/xxxxxxxxxx"

  routers:
    jellyfin_p-router:
      entrypoints:
        - "websecure"
      rule: "Host(`watch.myDomainOne.co`)"
      service: "jellyfin1"
      tls:
        certResolver: "le-dns_myDomainOne"
    7dtd-open-steam:
      rule: "Host(`7dtd.myDomainOne.co`)"
      middlewares:
        - "redirect-7dtd@file"
        - "7dtd-troubleshoot@file"
      service: "7dtd-steam"
    vaultwarden_web-router:
      entrypoints:
        - "websecure"
      rule: "Host(`vault.myDomainOne.co`)"
      service: "vaultwarden-web"
      tls:
        certResolver: "le-dns_myDomainOne"
    vaultwarden_ws-router:
      entrypoints:
        - "websecure"
      rule: "Host(`vault.myDomainOne.co`) && Path(`/notifications/hub`)"
      middlewares:
        - "bw-stripPrefix@file"
      service: "vaultwarden-ws"
      tls:
        certResolver: "le-dns_myDomainOne"
    freshrss-router:
      entrypoints:
        - "websecure"
      rule: "Host(`rss.myDomainOne.co`)"
      service: "freshrss-web"
      tls:
        certResolver: "le-dns_myDomainOne"
    nextcloud_web-router:
      entrypoints:
        - "websecure"
      rule: "Host(`docs.myDomainOne.co`)"
      service: "nextcloud-web"
      tls:
        certResolver: "le-dns_myDomainOne"
    nextcloud_collabora-router:
      entrypoints:
        - "websecure"
      rule: "Host(`collabora.myDomainOne.co`)"
      service: "nextcloud-collabora"
      tls:
        certResolver: "le-dns_myDomainOne"
    discord-invite:
      rule: "Host(`discord.myDomainOne.co`)"
      middlewares:
        - "redirect-discord@file"
      service: "discord"
    homer_r-router:
      entrypoints:
        - "websecure"
      rule: "Host(`myDomainTwo.com`)"
      service: "homer_r"
      tls:
        certResolver: "le-dns_myDomainTwo"
        domains:
          - main: myDomainTwo.com
            sans:
              - "*.myDomainTwo.com"
    jellyfin_r-router:
      entrypoints:
        - "websecure"
      rule: "Host(`pictures.myDomainTwo.com`)"
      service: "jellyfin_r"
      tls:
        certResolver: "le-dns_myDomainTwo"
        domains:
          - main: myDomainTwo.com
            sans:
              - "*.myDomainTwo.com"
    filebrowser_r-router:
      entrypoints:
        - "websecure"
      rule: "Host(`files.myDomainTwo.com`)"
      service: "filebrowser_r"
      tls:
        certResolver: "le-dns_myDomainTwo"
        domains:
          - main: myDomainTwo.com
            sans:
              - "*.myDomainTwo.com"
    ombi-router:
      entrypoints:
        - "websecure"
      rule: "Host(`request.myDomainOne.co`)"
      service: "ombi"
      tls:
        certResolver: "le-dns_myDomainOne"
  services:
    jellyfin1:
      loadBalancer:
        servers:
          - url: "http://192.168.1.55:8096"
    7dtd-steam:
      loadBalancer:
        servers:
          - url: "steam://connect/7d.myDomainOne.co:26900"
    vaultwarden-web:
      loadBalancer:
        servers:
          - url: "http://192.168.1.54:4480"
    vaultwarden-ws:
      loadBalancer:
        servers:
          - url: "http://192.168.1.54:3012"
    freshrss-web:
      loadBalancer:
        servers:
          - url: "http://192.168.1.54:8890"
    nextcloud-web:
      loadBalancer:
        servers:
          - url: "http://192.168.1.54:8888"
    nextcloud-collabora:
      loadBalancer:
        servers:
          - url: "http://192.168.1.54:8889"
    discord:
      loadBalancer:
        servers:
          - url: "https://discord.gg/xxxxxx"
    jellyfin_r:
      loadBalancer:
        servers:
          - url: "http://192.168.1.88:9055"
    homer_r:
      loadBalancer:
        servers:
          - url: "http://192.168.1.88:10000"
    filebrowser_r:
      loadBalancer:
        servers:
          - url: "http://192.168.1.88:9003"
    ombi:
      loadbalancer:
        servers:
          - url: "http://192.168.1.54:3579"
tls:
  options:
    default:
      cipherSuites:
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
      minVersion: VersionTLS12

There are generally no errors in the logs but I have found a few that happened at different times.

This error happened after I lost connection when the firefox extension vaultwarden tries to connect

time="2022-08-12T15:49:19-07:00" level=debug msg="Serving default certificate for request: \"\""
time="2022-08-12T15:49:19-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:33014: local error: tls: bad record MAC"
time="2022-08-12T16:09:41-07:00" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"\",\"RawPath\":\"\",\"OmitHost\":false,\"ForceQuery\":false,\"RawQuery\":\"access_token=LONG_TOKEN_HERE",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,en;q=0.5\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"Upgrade\"],\"Dnt\":[\"1\"],\"Origin\":[\"moz-extension://0b3e9d11-80f6-4619-ba3a-1d79d33f7b8a\"],\"Pragma\":[\"no-cache\"],\"Sec-Fetch-Dest\":[\"websocket\"],\"Sec-Fetch-Mode\":[\"websocket\"],\"Sec-Fetch-Site\":[\"same-origin\"],\"Sec-Websocket-Extensions\":[\"permessage-deflate\"],\"Sec-Websocket-Key\":[\"RdfjklPj8Lj3==\"],\"Sec-Websocket-Version\":[\"13\"],\"Upgrade\":[\"websocket\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Firefox/91.0\"],\"X-Forwarded-Host\":[\"vault.myDomainOne.co\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Prefix\":[\"/notifications/hub\"],\"X-Forwarded-Proto\":[\"wss\"],\"X-Forwarded-Server\":[\"ec239a541748\"],\"X-Real-Ip\":[\"xxx.xxx.xxx.xxx\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"vault.myDomainOne.co\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"xxx.xxx.xxx.xxx:36262\",\"RequestURI\":\"/?access_token=LONG_TOKEN_HERE",\"TLS\":null}"

Here is another that occasionally comes up:

time="2022-08-12T16:41:12-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:34988: tls: client offered only unsupported versions: [302 301]"

and another:

time="2022-08-12T17:39:55-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:50250: EOF"

These TLS errors only happen after connection is lost.

This may be a problem with my system but I suspect something with certificates is amiss. I have refreshed and recreated the .json files a hundred times to no avail. The fact that two of my apps work at all times suggests to me that there is something wrong with Traefik. These happen to be the latest two installed apps.

I will post more logs as I find them.

swiver · August 13, 2022, 4:08pm

Another refresh of the .json files and they are more consistent. myDomainThree comes online and spits out a few of these TLS errors after a restart of Traefik.

time="2022-08-13T08:31:21-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:32890: EOF"
time="2022-08-13T08:31:33-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:58896: EOF"
time="2022-08-13T08:31:33-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:58964: EOF"
time="2022-08-13T08:31:33-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:59018: EOF"
time="2022-08-13T08:31:33-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxxv:59044: tls: client requested unsupported application protocols ([http/0.9 http/1.0 spdy/1 spdy/2 spdy/3 h2c hq])"
time="2022-08-13T08:31:33-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:59058: tls: client requested unsupported application protocols ([hq h2c spdy/3 spdy/2 spdy/1 http/1.0 http/0.9])"
time="2022-08-13T08:31:34-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:59072: tls: client offered only unsupported versions: [302 301]"
time="2022-08-13T08:31:34-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:59090: EOF"
time="2022-08-13T08:31:34-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:59100: EOF"
time="2022-08-13T08:31:34-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:59248: EOF"
time="2022-08-13T08:31:34-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:59318: EOF"

swiver · August 14, 2022, 11:31pm

Here are some more recent errors:

time="2022-08-13T16:07:39-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:46706: tls: client used the legacy version field to negotiate TLS 1.3"
time="2022-08-13T17:30:35-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:55582: EOF"
time="2022-08-13T19:07:42-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:59918: remote error: tls: unknown certificate authority"
time="2022-08-13T20:27:19-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:46898: tls: client offered only unsupported versions: []"
time="2022-08-13T20:55:14-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:49660: tls: client offered only unsupported versions: [302 301]"
time="2022-08-13T21:00:52-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:45944: local error: tls: bad record MAC"
time="2022-08-13T21:38:20-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:54474: tls: client offered only unsupported versions: [301]"
time="2022-08-13T21:45:59-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:39024: tls: client offered only unsupported versions: []"
time="2022-08-13T22:27:44-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:22622: read tcp xxx.xxx.xxx.xxx:443->xxx.xxx.xxx.xxx:22622: read: connection reset by peer"
time="2022-08-13T22:39:51-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:49352: local error: tls: bad record MAC"
time="2022-08-13T23:53:52-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:65463: tls: client offered only unsupported versions: [301]"
time="2022-08-13T23:53:52-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:3961: tls: client offered only unsupported versions: []"
time="2022-08-13T23:53:52-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:24875: tls: unsupported SSLv2 handshake received"
time="2022-08-13T23:53:52-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:63069: tls: client offered only unsupported versions: [302]"
time="2022-08-13T23:53:53-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:56095: read tcp xxx.xxx.xxx.xxx:443->xxx.xxx.xxx.xxx:56095: read: connection reset by peer"
time="2022-08-14T01:30:54-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:35758: tls: no cipher suite supported by both client and server"
time="2022-08-14T01:30:54-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:35836: tls: client offered only unsupported versions: []"

time="2022-08-14T01:31:01-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:38020: tls: unsupported SSLv2 handshake received"

time="2022-08-14T01:31:02-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:38144: tls: unsupported SSLv2 handshake received"

time="2022-08-14T01:31:02-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:38210: tls: no cipher suite supported by both client and server"

time="2022-08-14T01:31:04-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:40044: tls: client offered only unsupported versions: [301]"

time="2022-08-14T01:31:12-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:42010: tls: no cipher suite supported by both client and server"

time="2022-08-14T01:31:23-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:44922: tls: client offered only unsupported versions: [302 301]"

time="2022-08-14T05:21:00-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:36042: EOF"
time="2022-08-14T05:34:36-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:39284: tls: client offered only unsupported versions: []"
time="2022-08-14T05:35:38-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:40282: tls: no cipher suite supported by both client and server"
time="2022-08-14T05:38:22-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:6594: EOF"
time="2022-08-14T05:38:23-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:14900: EOF"
time="2022-08-14T05:38:23-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:23470: tls: no cipher suite supported by both client and server"
time="2022-08-14T05:38:23-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:25552: tls: client requested unsupported application protocols ([http/0.9 http/1.0 spdy/1 spdy/2 spdy/3 h2c hq])"
time="2022-08-14T05:38:23-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:27698: tls: client requested unsupported application protocols ([hq h2c spdy/3 spdy/2 spdy/1 http/1.0 http/0.9])"
time="2022-08-14T05:38:23-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:29730: tls: client offered only unsupported versions: [302 301]"
time="2022-08-14T05:38:24-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:31828: read tcp xxx.xxx.xxx.xxx:443->xxx.xxx.xxx.xxx:31828: read: connection reset by peer"
time="2022-08-14T05:38:24-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:34078: EOF"
time="2022-08-14T05:38:25-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:42540: EOF"
time="2022-08-14T05:38:25-07:00" level=debug msg="http: TLS handshake error from xxx.xxx.xxx.xxx:50896: read tcp xxx.xxx.xxx.xxx:443->xxx.xxx.xxx.xxx:50896: read: connection reset by peer"

I thought what is happening might be related to the key type from these "unsupported versions" errors

I set the key type to RSA4096 and regenerated the acme-dns_....json files and it has not fixed the problem

Topic		Replies	Views
How to fix ACME error 400 connection refused Traefik v2 docker , docker-swarm , letsencrypt-acme , tcp	11	19051	November 16, 2020
ACME 400 Error: Connection refused - Works using Certbot Traefik v3 (latest) docker	1	48	October 23, 2024
Getting Error when trying to generate ACME certificates Traefik v2 docker , letsencrypt-acme	2	541	June 1, 2020
Traefik v2.8 letsencrypt renewal fails Traefik v2 docker , letsencrypt-acme	5	985	October 27, 2022
Traefik not accessible / acme error Traefik v2 docker , docker-swarm , dashboard-api , letsencrypt-acme	1	762	September 25, 2022

Intermittent Connection Failure

Related topics