I have a 3-node Nomad/Consul/Vault cluster.
Traefik is deployed in Nomad, and I want to load balance Nomad (and eventually will do the same for Consul/Vault).
Environment details:
Traefik: v3.1.7
Ubuntu: v24.10
Podman: v5.0.3
My issue is that I am getting 404 errors when I try to hit Nomad via Traefik.
I can query Consul for a list of discovered services and DNS querying is working without issue:
$ nslookup nomad.service.consul
Server: 127.0.0.53
Address: 127.0.0.53#53
Non-authoritative answer:
Name: nomad.service.consul
Address: 192.168.x.y
Name: nomad.service.consul
Address: 192.168.x.y
Name: nomad.service.consul
Address: 192.168.x.y
$ consul catalog services
consul
nomad
nomad-client
traefik
vault
I also validated that I can from the Traefik container check that the destination is open:
/ # traefik version
Version: 3.1.7
Codename: comte
Go version: go1.23.2
Built: 2024-10-28T10:41:06Z
OS/Arch: linux/arm64
/ #
/ # nc -vz 192.168.x.y 4646
192.168.x.y (192.168.x.y:4646) open
Here is my Traefik job definition:
job "traefik" {
datacenters = ["homelab"]
type = "service"
group "traefik" {
network {
port "http" {
static = 80
}
port "https" {
static = 443
}
}
service {
name = "traefik"
port = "https"
tags = [
"traefik.enable=true",
"traefik.http.routers.dashboard.rule=Host(`traefik.fqdn`)",
"traefik.http.routers.dashboard.service=api@internal",
"traefik.http.routers.dashboard.entrypoints=web,websecure",
"traefik.http.routers.dashboard.tls.certresolver=internal",
"traefik.http.routers.dashboard.tls=true",
]
check {
name = "alive"
type = "tcp"
port = "http"
interval = "10s"
timeout = "2s"
}
}
service {
name = "nomad"
port = "https"
tags = [
"traefik.enable=true",
"traefik.http.routers.nomad.rule=Host(`nomad.fqdn`)",
"traefik.http.routers.nomad.service=nomad",
"traefik.http.routers.nomad.entrypoints=web,websecure",
"traefik.http.routers.nomad.tls.certresolver=internal",
"traefik.http.routers.nomad.tls=true",
"traefik.http.services.nomad.loadbalancer.server.port=4646",
]
}
task "traefik" {
driver = "podman"
config {
image = "docker.io/library/traefik:v3.1.7"
ports = [
"http",
"https",
]
args = [
"--api.dashboard=true",
"--log.level=DEBUG",
"--accesslog=true",
# Consul integration
"--providers.consulcatalog=true",
"--providers.consulcatalog.exposedByDefault=false",
"--providers.consulcatalog.prefix=traefik",
"--providers.consulcatalog.endpoint.address=${NOMAD_IP_http}:8500",
# HTTP entrypoints
"--entrypoints.web.address=:${NOMAD_PORT_http}",
"--entrypoints.websecure.address=:${NOMAD_PORT_https}",
# Internal ACME/PKI
"--certificatesresolvers.internal.acme.caserver=https://ca.fqdn/acme/acme/directory",
"--certificatesresolvers.internal.acme.email=me@fqdn",
"--certificatesresolvers.internal.acme.storage=/local/internal.acme.json",
"--certificatesresolvers.internal.acme.tlschallenge=true",
"--certificatesresolvers.internal.acme.certificatesduration=24",
# Non-HTTP entrypoints
]
}
artifact {
source = "https://ca.fqdn/roots.pem"
mode = "file"
}
env {
LEGO_CA_CERTIFICATES = "/local/roots.pem"
}
resources {
cpu = 100
memory = 128
}
}
}
}
Here is a snippet of what Traefik's log looks like:
2024-11-11T19:15:13.833333245-05:00 stdout F 2024-11-12T00:15:13Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"routers":{"dashboard":{"entryPoints":["web","websecure"],"rule":"Host(`traefik.fqdn`)","service":"api@internal","tls":{"certResolver":"internal"}},"nomad":{"entryPoints":["web","websecure"],"rule":"Host(`nomad.fqdn`)","service":"nomad","tls":{"certResolver":"internal"}}},"services":{"nomad":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://192.168.x.y:4646"}]}},"traefik":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://192.168.x.y:443"}]}}}},"tcp":{},"tls":{},"udp":{}} providerName=consulcatalog
2024-11-11T19:15:13.833425246-05:00 stdout F 2024-11-12T00:15:13Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:127 > Skipping unchanged configuration providerName=consulcatalog
2024-11-11T19:15:28.832806157-05:00 stdout F 2024-11-12T00:15:28Z DBG github.com/traefik/traefik/v3/pkg/provider/consulcatalog/consul_catalog.go:287 > Filtering disabled item providerName=consulcatalog serviceName=consul
2024-11-11T19:15:28.834163221-05:00 stdout F 2024-11-12T00:15:28Z DBG github.com/traefik/traefik/v3/pkg/provider/consulcatalog/consul_catalog.go:287 > Filtering disabled item providerName=consulcatalog serviceName=nomad-client
2024-11-11T19:15:28.835410766-05:00 stdout F 2024-11-12T00:15:28Z DBG github.com/traefik/traefik/v3/pkg/provider/consulcatalog/consul_catalog.go:287 > Filtering disabled item providerName=consulcatalog serviceName=vault
2024-11-11T19:15:28.835784083-05:00 stdout F 2024-11-12T00:15:28Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"routers":{"dashboard":{"entryPoints":["web","websecure"],"rule":"Host(`traefik.fqdn`)","service":"api@internal","tls":{"certResolver":"internal"}},"nomad":{"entryPoints":["web","websecure"],"rule":"Host(`nomad.fqdn`)","service":"nomad","tls":{"certResolver":"internal"}}},"services":{"nomad":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://192.168.x.y:4646"}]}},"traefik":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://192.168.x.y:443"}]}}}},"tcp":{},"tls":{},"udp":{}} providerName=consulcatalog
2024-11-11T19:15:28.835868731-05:00 stdout F 2024-11-12T00:15:28Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:127 > Skipping unchanged configuration providerName=consulcatalog
2024-11-11T19:15:43.831264193-05:00 stdout F 2024-11-12T00:15:43Z DBG github.com/traefik/traefik/v3/pkg/provider/consulcatalog/consul_catalog.go:287 > Filtering disabled item providerName=consulcatalog serviceName=consul
2024-11-11T19:15:43.832552386-05:00 stdout F 2024-11-12T00:15:43Z DBG github.com/traefik/traefik/v3/pkg/provider/consulcatalog/consul_catalog.go:287 > Filtering disabled item providerName=consulcatalog serviceName=nomad-client
2024-11-11T19:15:43.833889894-05:00 stdout F 2024-11-12T00:15:43Z DBG github.com/traefik/traefik/v3/pkg/provider/consulcatalog/consul_catalog.go:287 > Filtering disabled item providerName=consulcatalog serviceName=vault
2024-11-11T19:15:43.834306971-05:00 stdout F 2024-11-12T00:15:43Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"routers":{"dashboard":{"entryPoints":["web","websecure"],"rule":"Host(`traefik.fqdn`)","service":"api@internal","tls":{"certResolver":"internal"}},"nomad":{"entryPoints":["web","websecure"],"rule":"Host(`nomad.fqdn`)","service":"nomad","tls":{"certResolver":"internal"}}},"services":{"nomad":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://192.168.x.y:4646"}]}},"traefik":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://192.168.x.y:443"}]}}}},"tcp":{},"tls":{},"udp":{}} providerName=consulcatalog
2024-11-11T19:15:43.834437990-05:00 stdout F 2024-11-12T00:15:43Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:127 > Skipping unchanged configuration providerName=consulcatalog
2024-11-11T19:15:58.835528971-05:00 stdout F 2024-11-12T00:15:58Z DBG github.com/traefik/traefik/v3/pkg/provider/consulcatalog/consul_catalog.go:287 > Filtering disabled item providerName=consulcatalog serviceName=nomad-client
2024-11-11T19:15:58.836710645-05:00 stdout F 2024-11-12T00:15:58Z DBG github.com/traefik/traefik/v3/pkg/provider/consulcatalog/consul_catalog.go:287 > Filtering disabled item providerName=consulcatalog serviceName=vault
2024-11-11T19:15:58.836741330-05:00 stdout F 2024-11-12T00:15:58Z DBG github.com/traefik/traefik/v3/pkg/provider/consulcatalog/consul_catalog.go:287 > Filtering disabled item providerName=consulcatalog serviceName=consul
2024-11-11T19:15:58.837119240-05:00 stdout F 2024-11-12T00:15:58Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"routers":{"dashboard":{"entryPoints":["web","websecure"],"rule":"Host(`traefik.fqdn`)","service":"api@internal","tls":{"certResolver":"internal"}},"nomad":{"entryPoints":["web","websecure"],"rule":"Host(`nomad.fqdn`)","service":"nomad","tls":{"certResolver":"internal"}}},"services":{"nomad":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://192.168.x.y:4646"}]}},"traefik":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://192.168.x.y:443"}]}}}},"tcp":{},"tls":{},"udp":{}} providerName=consulcatalog
2024-11-11T19:15:58.837196592-05:00 stdout F 2024-11-12T00:15:58Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:127 > Skipping unchanged configuration providerName=consulcatalog
2024-11-11T19:15:59.638960091-05:00 stdout F 192.168.x.y - - [12/Nov/2024:00:15:59 +0000] "GET / HTTP/1.1" 404 19 "-" "-" 11 "-" "-" 0ms
2024-11-11T19:16:00.743713080-05:00 stdout F 192.168.x.y - - [12/Nov/2024:00:16:00 +0000] "GET / HTTP/1.1" 404 19 "-" "-" 12 "-" "-" 0ms
2024-11-11T19:16:01.990451857-05:00 stdout F 192.168.x.y - - [12/Nov/2024:00:16:01 +0000] "GET / HTTP/1.1" 404 19 "-" "-" 13 "-" "-" 0ms
2024-11-11T19:16:03.148575495-05:00 stdout F 192.168.x.y - - [12/Nov/2024:00:16:03 +0000] "GET / HTTP/1.1" 404 19 "-" "-" 14 "-" "-" 0ms
2024-11-11T19:16:04.114611766-05:00 stdout F 192.168.x.y - - [12/Nov/2024:00:16:04 +0000] "GET / HTTP/1.1" 404 19 "-" "-" 15 "-" "-" 0ms
I am unsure of what is causing the issue and therefore how to rectify it.
Does anyone see anything wrong with my Traefik setup?
Thank you