Unable to get the ping service working on custom router & entryPoint

hreidar · March 13, 2020, 3:33pm

Hi, I'm having touble getting the ping service working in custom router and entryPoint. I'm using docker-compose and configuring everything using YAML (https://toolkit.site/format.html)

Here is my docker-compose file:

version: '3'

services:

  traefik:
    image: "traefik:v2.1.6"
    container_name: "traefik"
    command:
      - "--configFile=/etc/traefik/static.yaml"
    network_mode: "host"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "${PWD}/static.yaml:/etc/traefik/static.yaml:ro"
      - "${PWD}/conf/:/etc/traefik/conf/:ro"
      - "${PWD}/cert/:/etc/traefik/cert/:ro"
    restart: always
    logging:
      driver: "json-file"
      options:
        max-file: "2"
        max-size: "2m"

Here is my static file:

log:
  level: DEBUG

accessLog: {}

api:
  dashboard: true

entryPoints:
  http:
    address: ":80"
  https:
    address: ":443"
  k8s:
    address: ":6443"
  nipap:
    address: ":7000"
  ping:
    address: ":8081"
  metrics:
    address: ":8082"

ping:
  entryPoint: "ping"
  manualRouting: true

metrics:
  prometheus:
    addEntryPointsLabels: true
    addServicesLabels: true
    entryPoint: "metrics"
    buckets:
      - 0.1
      - 0.3
      - 0.5
      - 1.0
      - 5.0

providers:
  file:
    directory: "/etc/traefik/conf"
    watch: true
  #docker:
    #exposedbydefault: false

and finally my dynamic file:

http:
  ## routing ##
  routers:
    # attach middlewares to traefik api endpoint
    api:
      entryPoints:
        - http
      rule: Host(`inner-edge1.mydomain.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))
      service: api@internal
      middlewares:
        - acl-inner
#        - dash-auth

    # helth check service
    ping:
      entryPoints:
        - ping
      rule: Host(`inner-edge1.mydomain.com`) && PathPrefix(`/ping`)
      service: ping@internal
      middlewares:
        - acl-inner


    # traefik prometheus metrics endpoint
    metrics:
      entryPoints:
        - metrics
      rule: Host(`inner-edge1.mydomain.com`) && PathPrefix(`/metrics`)
      service: metrics
      middlewares:
        - acl-inner

  ## services ##
  services:
    metrics:
      loadBalancer:
        server:
          port: 8082

My requests @ http://inner-edge1.mydomain.com:8081/ping just timeout. Everything looks ok in the dashboard and netstat shows me that the server is listening on port 8081. Nothing appears in the access log as I'm sening the request. Any help is highly appreciated.

ldez · March 13, 2020, 5:38pm

version: '3.7'

services:

  traefik:
    image: traefik:v2.1.6
    command:
      - --log.level=INFO
      - --providers.docker.exposedbydefault=false
      
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
      - --entrypoints.ping.address=:8081
      - --entrypoints.metrics.address=:8083

      - --api
      - --ping.manualrouting
      - --metrics.prometheus.manualRouting
      - --metrics.prometheus.addEntryPointsLabels
      - --metrics.prometheus.addServicesLabels
      - --metrics.prometheus.buckets=0.1,0.3,0.5,1.0,5.0
    ports:
      - 80:80
      - 443:443
      - 8081:8081
      - 8083:8083
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    labels:
      traefik.enable: true

      # Dashboard
      traefik.http.routers.traefik.rule: Host(`traefik.localhost`)
      traefik.http.routers.traefik.entrypoints: websecure
      traefik.http.routers.traefik.service: api@internal
      traefik.http.routers.traefik.tls: true
      
      # Ping
      traefik.http.routers.traefik_ping.rule: Host(`traefik.localhost`) && PathPrefix(`/ping`)
      traefik.http.routers.traefik_ping.entrypoints: ping
      traefik.http.routers.traefik_ping.service: ping@internal

      # Metrics
      traefik.http.routers.traefik_metrics.rule: Host(`traefik.localhost`) && PathPrefix(`/metrics`)
      traefik.http.routers.traefik_metrics.entrypoints: metrics
      traefik.http.routers.traefik_metrics.service: prometheus@internal

$ curl http://traefik.localhost:8081/ping
OK

$ curl http://traefik.localhost:8083/metrics
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 1.6916e-05
go_gc_duration_seconds{quantile="0.25"} 0.000116391
go_gc_duration_seconds{quantile="0.5"} 0.000212668
go_gc_duration_seconds{quantile="0.75"} 0.000276674
go_gc_duration_seconds{quantile="1"} 0.003616942
go_gc_duration_seconds_sum 0.004239591
go_gc_duration_seconds_count 5
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 44
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.13.8"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 3.822288e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.296296e+07
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.445642e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 49269
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 0.009851411870272306
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 2.38592e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 3.822288e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 6.0489728e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 5.668864e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 19656
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 5.9301888e+07
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.6158592e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.5841217810599525e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 68925
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 13888
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 79152
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 114688
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 5.524688e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.214958e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 950272
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 950272
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.2286456e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 15
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 1.14
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 11
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 3.9686144e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.58412177878e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.69172992e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes -1
# HELP traefik_config_last_reload_failure Last config reload failure
# TYPE traefik_config_last_reload_failure gauge
traefik_config_last_reload_failure 0
# HELP traefik_config_last_reload_success Last config reload success
# TYPE traefik_config_last_reload_success gauge
traefik_config_last_reload_success 1.58412178e+09
# HELP traefik_config_reloads_failure_total Config failure reloads
# TYPE traefik_config_reloads_failure_total counter
traefik_config_reloads_failure_total 0
# HELP traefik_config_reloads_total Config reloads
# TYPE traefik_config_reloads_total counter
traefik_config_reloads_total 2
# HELP traefik_entrypoint_open_connections How many open connections exist on an entrypoint, partitioned by method and protocol.
# TYPE traefik_entrypoint_open_connections gauge
traefik_entrypoint_open_connections{entrypoint="metrics",method="GET",protocol="http"} 1

hreidar · March 14, 2020, 11:39am

Thank you for your reply @ldez. Your setup, using docker CLI and the Docker provider, seems to work as I expect my File provider config to work. I don't spot any difference except maybe the "traefik.enable: true" label which I can't find in the docs for File provider. Maybe I'm hitting some bug related to the File provider only.

ldez · March 14, 2020, 11:42am

There are no bug with the file provider, just mistakes in your configurations.

traefik.enable is only for Docker: https://docs.traefik.io/v2.1/providers/docker/#exposedbydefault

As you are using Docker I recommend to not use the file provider to create the routing if you can avoid.

hreidar · March 14, 2020, 12:03pm

Hi, and thanks again for your help.

Can you please point out the mistake I'm making?

Why, shouldn't I use the file provider? I'm trying to make things easy for my colleges, configure Traefik so that they only need to copy a file for each service they like to expose to our users. This way we can also use git for configuration tracking and storage.

ldez · March 14, 2020, 12:09pm

The main goal of Traefik is to manage dynamic architecture:

a container appears, Traefik detect it and create the configuration (based on labels)
a container disappears, Traefik detect it and remove the configuration

The file provider breaks this dynamic behavior.

The file provider, for the routing, has been introduced for only old architecture (VMs, bare metal servers, ...)

You can use several docker-compose files, shared them and use git with them.

I will give you a fix but I still recommend to not use the file provider to create the routing when you are using Docker.

docker-compose.yml

version: '3'

services:

  traefik:
    image: traefik:v2.1.6
    container_name: traefik
    ports:
      - 80:80
      - 443:443
      - 6443:6443
      - 7000:7000
      - 8081:8081
      - 8082:8082
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "${PWD}/static.yaml:/etc/traefik/traefik.yaml:ro"
      - "${PWD}/conf/:/etc/traefik/conf/:ro"
      - "${PWD}/cert/:/etc/traefik/cert/:ro"
    restart: always
    logging:
      driver: "json-file"
      options:
        max-file: "2"
        max-size: "2m"

static.yaml

log:
  level: INFO

accessLog: {}

api:
  dashboard: true

entryPoints:
  http:
    address: ":80"
  https:
    address: ":443"
  k8s:
    address: ":6443"
  nipap:
    address: ":7000"
  ping:
    address: ":8081"
  metrics:
    address: ":8082"

ping:
  manualRouting: true

metrics:
  prometheus:
    addEntryPointsLabels: true
    addServicesLabels: true
    manualRouting: true
    buckets:
      - 0.1
      - 0.3
      - 0.5
      - 1.0
      - 5.0

providers:
  file:
    directory: "/etc/traefik/conf"
    watch: true

bad-thing.yaml

http:

  ## routing ##
  routers:
    api:
      entryPoints:
        - http
      rule: Host(`traefik.localhost`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))
      service: api@internal
      middlewares:
        - acl-inner
        # - dash-auth

    # helth check service
    ping:
      entryPoints:
        - ping
      rule: Host(`traefik.localhost`) && PathPrefix(`/ping`)
      service: ping@internal
      middlewares:
        - acl-inner

    # traefik prometheus metrics endpoint
    metrics:
      entryPoints:
        - metrics
      rule: Host(`traefik.localhost`) && PathPrefix(`/metrics`)
      service: prometheus@internal
      middlewares:
        - acl-inner

hreidar · March 14, 2020, 12:39pm

Hi, I get you point, I have a Consul cluster running, maybe I have to look into using that. This setup is running on a VM though in an on-prem environment.

So I seem to be blind as I'm not seeing a significant difference.

You added ports to the docker-compose config instead of my network_mode: "host" which is something I will do after I have finalized my setup.

You also dropped the entryPoint: "ping" from the ping config in the static config.

And you changed the host rules to traefik.localhost

Am I missing something else ?

ldez · March 14, 2020, 12:48pm

--- old.yaml	2020-03-14 13:45:21.077476025 +0100
+++ new.yaml	2020-03-14 13:45:14.457466992 +0100
@@ -1,5 +1,5 @@
 log:
-  level: DEBUG
+  level: INFO
 
 accessLog: {}
 
@@ -21,14 +21,13 @@
     address: ":8082"
 
 ping:
-  entryPoint: "ping"
   manualRouting: true
 
 metrics:
   prometheus:
     addEntryPointsLabels: true
     addServicesLabels: true
-    entryPoint: "metrics"
+    manualRouting: true
     buckets:
       - 0.1
       - 0.3
@@ -40,5 +39,3 @@
   file:
     directory: "/etc/traefik/conf"
     watch: true
-  #docker:
-    #exposedbydefault: false

--- old.yaml	2020-03-14 13:47:48.947689437 +0100
+++ new.yaml	2020-03-14 13:47:11.157632930 +0100
@@ -1,7 +1,7 @@
 http:
+
   ## routing ##
   routers:
-    # attach middlewares to traefik api endpoint
     api:
       entryPoints:
         - http
@@ -20,19 +20,11 @@
       middlewares:
         - acl-inner
 
-
     # traefik prometheus metrics endpoint
     metrics:
       entryPoints:
         - metrics
       rule: Host(`inner-edge1.mydomain.com`) && PathPrefix(`/metrics`)
-      service: metrics
+      service: prometheus@internal
       middlewares:
         - acl-inner
-
-  ## services ##
-  services:
-    metrics:
-      loadBalancer:
-        server:
-          port: 8082

As you don't provide the configuration of your middlewares, I can not be sure there are no errors in this part.

I changed the host to traefik.localhost to be able to test it easily.

hreidar · March 14, 2020, 1:18pm

This is strange, 8082/metrics works fine but I don't get any responce from 8081/ping but the config seems the same, in most parts.

I commented out he middleware. It was just an ipWhitelist with our network which my client is currently on.

static.yaml now looks like this:

log:
  level: DEBUG

accessLog: {}

api:
  dashboard: true

entryPoints:
  http:
    address: ":80"
  https:
    address: ":443"
  k8s:
    address: ":6443"
  nipap:
    address: ":7000"
  ping:
    address: ":8081"
  metrics:
    address: ":8082"

ping:
  manualRouting: true

metrics:
  prometheus:
    addEntryPointsLabels: true
    addServicesLabels: true
    manualRouting: true
    buckets:
      - 0.1
      - 0.3
      - 0.5
      - 1.0
      - 5.0

providers:
  file:
    directory: "/etc/traefik/conf"
    watch: true

and traefik-services.yaml like this:

http:
  ## routing ##
  routers:
    # attach middlewares to traefik api endpoint
    api:
      entryPoints:
        - "http"
      rule: "PathPrefix(`/api`) || PathPrefix(`/dashboard`)"
      service: "api@internal"
#      middlewares:
#        - "acl-inner"
#        - "dash-auth"

    # helth check service
    ping:
      entryPoints:
        - "ping"
      rule: "PathPrefix(`/ping`)"
      service: "ping@internal"
#      middlewares:
#        - "acl-inner"


    # traefik prometheus metrics endpoint
    metrics:
      entryPoints:
        - "metrics"
      rule: "PathPrefix(`/metrics`)"
      service: "prometheus@internal"
#      middlewares:
#        - "acl-inner"

Again, if I change the entryPoints of the ping router to http the service works on port 80

hreidar · March 14, 2020, 1:44pm

@ldez, thank you for your time. I detected a ACTIVE firewall on the host which explains a lot . I had previously deactivated it when I started working on this project.

Topic		Replies	Views
Offering ping service on the same entrypoint as the api/dashboard? Traefik v2 docker , ping	1	475	April 19, 2021
Expose ping to external load balancer Traefik v2 ping	10	6162	September 28, 2020
Ping over TLS - Do not understand how to implement this Traefik v2 docker , ping	0	2031	November 22, 2020
Traefik v2 /ping returning 404 Traefik v2 kubernetes-ingress , ping	4	2505	September 28, 2020
Ping endpoint working but docker healthcheck fails Traefik v2 docker , ping	4	5005	March 1, 2021

Unable to get the ping service working on custom router & entryPoint

Related topics