Traefik not working with podman and nomad

Traefik Traefik v3 (latest)
1

I'm tring to use traefik to route ports with podman images (podman images containing Traefik and Rstudio Server) and nomad, but cannot get traefik to route port 9999 to 8787 (rstudio server). I'm first testing this out on a laptop with http.

I get a 404 when I visit http://127.0.0.1:9999 , but can access RStudio server at http://127.0.0.1:8787

I can see an error in the consul log file: error="No known Consul servers"

I see this in the Traefik log:
2024-11-05T09:47:00.178805340+07:00 stdout F 2024-11-05T02:47:00Z ERR KV connection error, retrying in 261.040918ms error="KV store connection error: Unexpected response code: 500 (No cluster leader)" providerName=consul

I've tried to modify the nomad and consul configuration files, but no success as yet.

I can see the traefik and rstudio server podman services running within the Nomad web interface at http://127.0.0.1:4646.

I'd be grateful for any suggestions.

Here are some relevant configuration details.

  1. Nomad job file:
job "traefik-rstudio" {
datacenters = ["dc1"]
type = "service"

group "traefik-group" {
count = 1

network {
port "web" {
static = 9999
}
port "api" {
static = 8080
}
}

task "traefik" {
driver = "podman"

config {
image = "docker.io/library/traefik:v3.1.6"
args = [
"--entrypoints.web.address=:9999",
"--providers.nomad=true",
"--providers.consul=true",
"--providers.consul.endpoints=127.0.0.1:8500",
"--providers.consul.rootKey=traefik",
"--api.insecure=true"

]
ports = ["web"]
network_mode = "host"
}

resources {
cpu    = 500
memory = 256
}
}
}

group "rstudio-group" {
count = 1

network {
port "http" {
static = 8787
}
}

service {
name = "rstudio"
port = "http"
tags = [
"traefik.enable=true",
"traefik.docker.network=test_net2",
"traefik.http.routers.rstudio.entrypoints=web",
"traefik.http.routers.rstudio.rule=Host(`127.0.0.1`)",
"traefik.http.middlewares.rstudio.stripprefix.prefixes=/",
"traefik.http.routers.rstudio.middlewares=rstudio@docker",
"traefik.http.services.rstudio.loadbalancer.server.port=8787",
"traefik.http.services.rstudio.loadbalancer.sticky=true",
"traefik.http.services.rstudio.loadbalancer.sticky.cookie.name=stickycookie",
"traefik.http.services.rstudio.loadbalancer.sticky.cookie.secure=false",
"traefik.http.services.rstudio.loadbalancer.sticky.cookie.httpOnly=true",
"traefik.http.routers.consul.entrypoints=http"
]
provider = "consul"
check {
name     = "alive"
type     = "http"
path     = "/"
interval = "10s"
timeout  = "2s"
}
}

task "rstudio-server" {
driver = "podman"

config {
image = "docker.io/library/rstudio:r4.4.1"
ports = ["http"]
}

resources {
cpu    = 500
memory = 4024
}
}
}
}
  1. /etc/consul.d/consul.hcl:

datacenter = "dc1"
data_dir = "/opt/consul"
server = true
bind_addr = "0.0.0.0" # Listen on all IPv4
advertise_addr = "127.0.0.1"
  1. /etc/systemd/system/nomad.service
[Unit]
Description=Nomad Agent
After=network.target

[Service]
ExecStart=/usr/bin/nomad agent -config /etc/nomad.d
Restart=on-failure

[Install]
WantedBy=multi-user.target
  1. /etc/nomad.d/nomad.hcl
data_dir = "/opt/nomad"

bind_addr = "127.0.0.1"

# Enable the server mode
server {
enabled = true
bootstrap_expect = 1
}

# Enable the client mode
client {
enabled = true
network_interface = "lo"
}

# Specify the plugin directory
plugin_dir = "/opt/nomad/plugins"

# Configure the Podman driver plugin
plugin "nomad-driver-podman" {
config {
socket_path = "unix:///run/podman/podman.sock"
}
}

advertise {
http = "127.0.0.1:4646"
rpc  = "127.0.0.1:4647"
serf = "127.0.0.1:4648"
}

# Enable the raw_exec plugin to run local commands
plugin "raw_exec" {
config {
enabled = true
}
}
  1. Software versions
    Nomad version: v1.9.1
    Podman version: 3.4.4
    Traefik version: 3.1.6 (also tried 3.1.0)
    OS: Ubuntu 22.04

I run the above set up as follows:

sudo systemctl restart podman
sudo systemctl restart nomad
sudo nohup consul agent -config-dir=/etc/consul.d > /var/log/consul.log 2>&1 &
nomad job run nomad.rstudio_and_traefik

The services within the Nomad web interface (http://127.0.0.1:4646/ui/jobs) are running / healthy.

2

According to error message, Traefik has an issue with the consul connection.

I don’t know about podman, nomad and consul, but with regular Docker the Traefik instance would usually not be able to reach another service at 127.0.0.1, as that is only container-internal localhost, not node localhost, as containers are made for isolation.