Traefik Proxy Integrates with Hashicorp Nomad | Traefik Labs

[Blog]

We are thrilled to announce the full integration of the new Nomad built-in Service Discovery with Traefik Proxy. This is a first-of-its-kind ingress integration that simplifies ingress in HashiCorp Nomad. Utilizing Nomad directly with Traefik Proxy has never been so easy!

In early May, Hashicorp announced Nomad Version 1.3. Among other updates, it also includes a nice list of improvements on usability and developer experience. Before this release, when using service discovery with Nomad, Traefik Proxy users had to use Hashicorp Consul and Nomad side-by-side in order to benefit from Traefik Proxy’s famous automatic configuration. Now, Nomad has a simple and straightforward way to use service discovery built-in. This improves direct usability a lot! Not only in simple test environments but also on the edge.

With this great new addition, the major question remained: How can you utilize the new Nomad Service Discovery directly with an ingress? To provide you with an answer, we worked closely together with Hashicorp to get a first implementation of those new APIs in Traefik Proxy.

This is a big step, and we are very excited to be the first ingress and reverse proxy that integrates natively with the new Nomad Service Discovery. Traefik Proxy (and soon Enterprise) users can now utilize the new Nomad provider in Traefik Proxy, to unlock its full potential directly with Nomad, as it was possible previously only with Consul.

The main driving force behind our Nomad integration in Traefik was always you, our community. We know that our Nomad community is really large and active, and uses Traefik in a lot of different ways. Therefore, it was only logical to provide our community with this additional new integration.

Let’s dig in and see how this integration works.

Set up Traefik Proxy with the new Nomad Service Discovery

Utilizing a new provider with Traefik Proxy is simple and straightforward. First, make sure that you have Nomad set up with the required version. Remember, the minimum required version is 1.3.0.

nomad -v
Nomad v1.3.1 (2b054e38e91af964d1235faa98c286ca3f527e56)

Once that is verified, we can deploy Traefik Proxy. This new integration requires at least version 2.8-rc1. With that version, you can enable the new provider as seen below:

job "traefik" {
  datacenters = ["dc1"]
  type        = "service"
  group "traefik" {
    count = 1
    network {
      port  "http"{
         static = 80
      }
      port  "admin"{
         static = 8080
      }
    }
    service {
      name = "traefik-http"
      provider = "nomad"
      port = "http"
    }
    task "server" {
      driver = "docker"
      config {
        image = "traefik:2.8-rc1"
        ports = ["admin", "http"]
        args = [
          "--api.dashboard=true",
          "--api.insecure=true", ### For Test only, please do not use that in production
          "--entrypoints.web.address=:${NOMAD_PORT_http}",
          "--entrypoints.traefik.address=:${NOMAD_PORT_admin}",
          "--providers.nomad=true",
          "--providers.nomad.endpoint.address=http://192.168.178.39:4646" ### IP to your nomad server 
        ]
      }
    }
  }
}

Once Traefik Proxy is up and running with the dashboard enabled (for test purposes), you can verify that the provider is correctly enabled:

With that verified, you can now deploy your first service. In order to enable the new Service Discovery for a service, you need to enable it in the service stanza. In the service stanza there is now a provider option which needs to be set to nomad. Below you see a full example using the sample Traefik Proxy application whoami.

job "whoami" {
  datacenters = ["dc1"]
  type = "service"
  group "demo" {
    count = 1
    network {
       port "http" {
         to = 80
       }
    }
    service {
      name = "whoami-demo"
      port = "http"
      provider = "nomad"
      tags = [
        "traefik.enable=true",
        "traefik.http.routers.http.rule=Host(`whoami.nomad.localhost`)",
      ]
    }
    task "server" {
      env {
        WHOAMI_PORT_NUMBER = "${NOMAD_PORT_http}"
      }
      driver = "docker"
      config {
        image = "traefik/whoami"
        ports = ["http"]
      }
    }
  }
}

With the service deployed, you should be able to see that Traefik Proxy picked it up quickly in the dashboard as well.

Testing it with a simple curl (make sure to set a host entry!) should work like a charm from here:

curl -v http://whoami.nomad.localhost
*   Trying 10.27.71.49:80...
* Connected to whoami.nomad.localhost (10.27.71.49) port 80 (#0)
> GET / HTTP/1.1
> Host: whoami.nomad.localhost
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Length: 365
< Content-Type: text/plain; charset=utf-8
< Date: Mon, 06 Jun 2022 11:56:11 GMT
<
Hostname: a65ebc9c1731
IP: 127.0.0.1
IP: 172.17.0.3
RemoteAddr: 172.17.0.1:60030
GET / HTTP/1.1
Host: whoami.nomad.localhost
User-Agent: curl/7.79.1
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 172.17.0.1
X-Forwarded-Host: whoami.nomad.localhost
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: 08f885acac7d
X-Real-Ip: 172.17.0.1

Conclusion

The new native Service Discovery in Nomad really does work seamlessly. With this integration, delivering load balancing, dynamic routing configuration, and ingress traffic routing become easier than ever. Check out the Traefik Proxy 2.8 Release Candidate and the Nomad 1.3 release notes. You’re also welcome to head over to our Community Forum to share your feedback and ask any questions.

If you want to learn more, register for our upcoming webinar “Simplify Networking in Nomad with New Traefik Integration” on June 30, 2022. You can also join the HashiConf Europe session “Nomad: Past, Present, and Future,” on Tuesday, June 21, and visit the Hashicorp Nomad booth for a demo.

---

This is a companion discussion topic for the original entry at https://traefik.io/blog/traefik-proxy-fully-integrates-with-hashicorp-nomad/

[fireharp]

Somehow I have problems with this setup when working through vagrant VM.

Traefik dash seem to work nice on vagrant

Then connection's reset on the host machine.
I've tested the same vagrant config with some simple server and it's working OK

This makes me think that connection reset actually happens somewhere in between Nomad/Traefik.

Vagrantfile, traefik job in details below

Files

traefic.hcl

job "traefik" {
  datacenters = ["dc1"]
  type        = "service"

  group "traefik" {
    count = 1

    network {
      port  "http"{
         static = 80
      }
      port  "admin"{
         static = 8080
      }
    }

    service {
      name = "traefik-http"
      provider = "nomad"
      port = "http"
    }

    task "server" {
      driver = "docker"
      config {
        image = "traefik:2.8.0-rc1"
        ports = ["admin", "http"]
        args = [
          "--providers.nomad=true",
          "--providers.nomad.endpoint.address=http://10.0.2.15:4646", ### IP to your nomad server 
          ### For Test only, please do not use that in production
          "--api.dashboard=true",
          "--api.insecure=true", 
          "--entrypoints.web.address=0.0.0.0:${NOMAD_PORT_http}",
          "--entrypoints.traefik.address=0.0.0.0:${NOMAD_PORT_admin}"
        ]
      }
    }
  }
}

Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :

$script = <<SCRIPT
echo "Installing Docker..."
sudo apt-get update
sudo apt-get remove docker docker-engine docker.io
echo '* libraries/restart-without-asking boolean true' | sudo debconf-set-selections
sudo apt-get install apt-transport-https ca-certificates curl software-properties-common -y
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg |  sudo apt-key add -
sudo apt-key fingerprint 0EBFCD88
sudo add-apt-repository \
      "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
      $(lsb_release -cs) \
      stable"
sudo apt-get update
sudo apt-get install -y docker-ce
# Restart docker to make sure we get the latest version of the daemon if there is an upgrade
sudo service docker restart
# Make sure we can actually use docker as the vagrant user
sudo usermod -aG docker vagrant
sudo docker --version

# Packages required for nomad & consul
sudo apt-get install unzip curl vim -y
# some utility
sudo apt install nodejs npm

echo "Installing Nomad..."
NOMAD_VERSION=1.3.1
cd /tmp/
curl -sSL https://releases.hashicorp.com/nomad/${NOMAD_VERSION}/nomad_${NOMAD_VERSION}_linux_amd64.zip -o nomad.zip
unzip nomad.zip
sudo install nomad /usr/bin/nomad
sudo mkdir -p /etc/nomad.d
sudo chmod a+w /etc/nomad.d


echo "Installing Consul..."
CONSUL_VERSION=1.12.2
curl -sSL https://releases.hashicorp.com/consul/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_linux_amd64.zip > consul.zip
unzip /tmp/consul.zip
sudo install consul /usr/bin/consul
(
cat <<-EOF
  [Unit]
  Description=consul agent
  Requires=network-online.target
  After=network-online.target

  [Service]
  Restart=on-failure
  ExecStart=/usr/bin/consul agent -dev
  ExecReload=/bin/kill -HUP $MAINPID

  [Install]
  WantedBy=multi-user.target
EOF
) | sudo tee /etc/systemd/system/consul.service
sudo systemctl enable consul.service
sudo systemctl start consul

for bin in cfssl cfssl-certinfo cfssljson
do
  echo "Installing $bin..."
  curl -sSL https://pkg.cfssl.org/R1.2/${bin}_linux-amd64 > /tmp/${bin}
  sudo install /tmp/${bin} /usr/local/bin/${bin}
done
nomad -autocomplete-install

SCRIPT

Vagrant.configure(2) do |config|
  config.vm.box = "bento/ubuntu-20.04" # 20.04 LTS
  config.vm.hostname = "nomad"
  config.vm.provision "shell", inline: $script, privileged: false

  # config.vm.synced_folder ".", "/nomad-demo/"

  # Expose the nomad api and ui to the host
  config.vm.network "forwarded_port", guest: 4646, host: 4646, auto_correct: true, host_ip: "127.0.0.1"
  # Expose the traefik to the host
  config.vm.network "forwarded_port", guest: 8080, host: 8080, auto_correct: true, host_ip: "127.0.0.1"
  config.vm.network "forwarded_port", guest: 80, host: 80, auto_correct: true, host_ip: "127.0.0.1"

  # Increase memory for Parallels Desktop
  config.vm.provider "parallels" do |p, o|
    p.memory = "1024"
  end

  # Increase memory for Virtualbox
  config.vm.provider "virtualbox" do |vb|
        vb.memory = "1024"
  end

  # Increase memory for VMware
  ["vmware_fusion", "vmware_workstation"].each do |p|
    config.vm.provider p do |v|
      v.vmx["memsize"] = "1024"
    end
  end
end

[fireharp]

It's probably Traefic docker binds itself onto localhost. Should be 0.0.0.0 for vagrant setup.

Not sure how to manage this right off.

[fireharp]

Another thing with this example.

If I deploy it straight forward (no vagrant).

It still fails on last step of checking http://whoami.nomad.localhost/

Traefik seem to pick it up correctly

curl/ping don't recognize the host

(base) ➜  nomad-demo ping whoami.nomad.localhost  
ping: cannot resolve whoami.nomad.localhost: Unknown host
(base) ➜  nomad-demo curl -v http://whoami.nomad.localhost    
* Could not resolve host: whoami.nomad.localhost
* Closing connection 0
curl: (6) Could not resolve host: whoami.nomad.localhost

browser responds with 502 Bad Gateway

[daoyuanzhai]

(post deleted by author)

[joh4n]

Hi,
I have tried following the tutorial with the following "modifications" to get it to work in a minimal dev setup:

• used nomad v1.3.5
• start nomad agent with: sudo nomad agent -dev -bind 0.0.0.0 -log-level INFO
• set treafik image to: traefik:2.8 (also tried traefik:2.9)
• tried various versions of the line: "--providers.nomad.endpoint.address=http://192.168.178.39:4646" , including commenting it out, setting it to the default value=1127.0.0.1:46461

When running through the tutorial traefik does not manage to connect to the nomad service discovery, nomad alloc logs f671f.... gives:

time="2022-09-22T15:46:27Z" level=error msg="Provider connection error failed to load initial nomad services: Get "http://127.0.0.1:4646/v1/services\": dial tcp 127.0.0.1:4646: connect: connection refused, retrying in 15.401737866s" providerName=nomad

So traefik never manages to detect the whoami-demo service. Any ideas of what the issue can be? / what I should set providers.nomad.endpoint.address for local development?

It seems related to

• Traefik unable to connect to Nomad Service Provider
  But going through it does not provide any answers

(I asked the same question here: Traefik unable to connect to Nomad Service Provider - #7 by joh4n)