502 Bad Gateways

Our Web service is working on port 80 but not working on 443 port.
Internally we are using docker overlay network.
When I run application I am getting error as - 502 bad gateway an existing connection was forcibly closed by the remote host.
I can use Docker Exec from the Traefik container and can successfully connect to the IIS site through the overlaynetwork on port 443 successfully.
Note- We are using self signed certificate for testing purposes.

job "traefik" {
datacenters = ["dc1"]
type = "system"

update {
healthy_deadline = "30m"
progress_deadline = "40m"
}

group "traefik" {
count = 1

network {
  port "http" {
    static = 80
  }
  port "https" {
    static = 443
  }
  port  "admin"{
    static = 8080
  }
}

task "traefik" {
  driver = "docker"    

  service {      
    provider = "nomad"
    port = "http"

    tags = [
        "traefik.enable=true",
    ]

    check {
      name     = "alive"
      type     = "tcp"
      port     = "http"
      interval = "10s"
      timeout  = "2s"
    }
  }	  

  
  config {
    image = "traefik:latest"
    network_mode = "Overlaynetwork"             
    image_pull_timeout = "30m"         
    ports = ["admin", "http", "https"]
    args = [
      "--accesslog=true",
      "--api=true",
      "--api.dashboard=true",
      "--api.insecure=true", ### For Test only, please do not use that in production
      "--metrics=true",
      "--metrics.prometheus=true",
      "--ping=true",
      "--entrypoints.web.address=:80",
      "--entrypoints.websecure.address=:443",
      "--entrypoints.traefik.address=:8080",
      "--providers.nomad=true",
      "--providers.nomad.endpoint.address=http://*.*.*.*:*", ### IP to your nomad server 
	  "--serversTransport.insecureSkipVerify=true",
	  "--log.level=DEBUG"
    ]
    
    auth {
      username = ""
      password = ""
    }
  }
}

}
}

Nomad Web Job -

job "dotnetpocKeyVault" {
type = "service"
update {
healthy_deadline = "60m"
progress_deadline = "70m"
}
datacenters = ["dc1"]
group "webs" {
count = 1
network {
port "https" {
to = 443
}
}

	task "dotnet" {
		driver = "docker"
		service {
			provider = "nomad"
			port = "https"
			tags = [
					"traefik.enable=true",
					"traefik.http.routers.nomadpoc.rule=Host(`abc.com`)", 
					"traefik.http.routers.nomadpoc.entrypoints=https",
					"traefik.http.routers.nomadpoc.tls =true"
					
			]
			check {
				type = "tcp"
				port = "https"
				interval = "10s"
				timeout = "2s"
			}
		}
		
		
		template {

				data =  <<EOF

{{plugin "powershell" }}{{file "D:/PowershellScript/SSLCertificate.pfx" }}
EOF
destination = "local\SSLCertificate.pfx"
change_mode="noop"
}

		# Configuration is specific to each driver.
							
		config {
			network_mode = "Overlaynetwork"
			image = "poc:latest"
    			ports = ["https"]        
			image_pull_timeout = "30m"
			auth {
      				username = ""
      				password = ""
    			 }
				 
		}
		
		resources {
			cpu = 1024 # MHz
			memory = 1024 # MB
		}
		
	}
	
}

}

Use 3 backticks in front and after the code to format it, or select the code and press the </> button.

Share your full Traefik static and dynamic config, and docker-compose.yml if used.

Enable Traefik debug log and check the Traefik dashboard.

Gateway problems are usually an issue with the Docker network:

  1. target service is connected to multiple networks and provider.docker.network is not set or incorrect
  2. Docker network is established within docker-compose.yml and name is not set, then the name is automatically extended by the project name and doesn’t match anymore
  3. Docker overlay network is used and the MTU is not adapted to a smaller value that fits inside a VLAN used
job "dotnetpocKeyVault" {
	type = "service"
	update {
		healthy_deadline = "60m"
		progress_deadline = "70m"
	}
	datacenters = ["dc1"]
	group "webs" {
		count = 1
		network {
			port "https" {
				to = 443
			}
		}
		
		task "dotnet" {
			driver = "docker"
			service {
				provider = "nomad"
				port = "https"
				tags = [
						"traefik.enable=true",
						"traefik.http.routers.nomadpoc.rule=Host(`abc.com`)", 
						"traefik.http.routers.nomadpoc.entrypoints=websecure",
						"traefik.http.routers.nomadpoc.tls =true"
						
				]
				check {
					type = "tcp"
					port = "https"
					interval = "10s"
					timeout = "2s"
				}
			}
			template {
      					destination = "${NOMAD_SECRETS_DIR}/env.vars"
      					env         = true
      					data        = <<EOF
      						{{- with nomadVar "nomad/jobs" -}}
      						user = {{.NomadPOCArtifactoryID}}
      						password = {{.NomadPOCArtifactoryPassword}}
      						{{- end -}}
      					EOF
    			}
			
					template {

					data =  <<EOF
{{plugin "powershell"  "& D:\\PowershellScript\\ReadStoreCertificate.ps1"}}{{file "D:/PowershellScript/TlsCertificate.pfx" }}
						EOF
        				destination   = "local\\TlsCertificate.pfx"
						change_mode="noop"
        			}			
			
							
			config {
				network_mode = "Overlaynetwork"
				image = "poc:latest"
        			ports = ["https"]        
				image_pull_timeout = "30m"
				auth {
          				username = "${user}"
          				password = "${password}"
        			 }
					 
			}
			
			resources {
				cpu = 1024 # MHz
				memory = 1024 # MB
			}
			
		}
		
	}
}
``

1

@bluepuma77 My logs are -
502 Bad Gateway error="dial tcp ...:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond."

Service selected by WRR: 9011a4d279bb7890

502 Bad Gateway error="read tcp ...:1000->...:443: wsarecv: An existing connection was forcibly closed by the remote host."

DBG tlsmanager.go:220 > Serving default certificate for request: ""

Not sure what is happening here. IP 1.1.1.1 is an Internet DNS server. Which is normally connected via port 53.

Either your firewall is blocking regular DNS traffic or you are (mis-)using public IPs for a private network, then routing becomes an issue. (Wikipedia)

@bluepuma77 while pasting logs on Traefik forum I changed my container Ip address with 1.1.1.1 (I
Just wanted to put some dummy IP address while posting . I was not aware it is Internet DNS server Ip address)

Use docker inspect to get the Docker network IP of the target service. Use docker exec -it <name> sh to go into the Traefik container. Try ping 1.2.3.4 and wget http://1.2.3.4 to see if the target is reachable.

@bluepuma77 yes, the target container is reachable through traefik , but when trying to access it through browser getting Bad getaway error .

[Note- Internally we are using docker overlay network. I deployed Traefik and target web job on windows container and I am using self signed certificate for testing purposes ]

@bluepuma77 any suggestion

No idea. You have a maximum complicated setup: windows, nomad, custom certs, IIS, swarm...

Not sure if the additional space does anything:
"traefik.http.routers.nomadpoc.tls =true".

Check Traefik dashboard and Traefik debug log.

Make sure if you use a VLAN that the MTU of the Docker overlay network is set accordingly. I debugged many days because its hard to diagnose: simple ping works, even whoami, as long as the response is below ~1400 bytes.

@bluepuma77 ..In traefik dashboard I can see URL as http for HTTPS connection . is it correct ?[Note- In dashboard I an see TLS enebled only on URL I can see http]
Http

This is my test dashboard with http/s routers

can you please check the url by clicking on rule . is it Https or http