Marathon and traefik 2.1 not reading /etc/hosts of container resulting in 504 Gateway timeout


What did you do?

I have deployed the following app on marathon:

  "id": "/whoami",
  "cpus": 0.1,
  "mem": 256.0,
  "instances": 3,
  "labels": {
    "traefik.enable": "true",
    "traefik.http.routers.whoami.rule": "Host(``)",
    "traefik.http.routers.whoami.entrypoints": "web-secure",
    "traefik.http.routers.whoami.tls.certresolver": "letsencryptStaging",
    "[0].main": "",
    "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme": "https",
    "traefik.http.routers.redirs.rule": "hostregexp(`{host:.+}`)",
    "traefik.http.routers.redirs.entrypoints": "web",
    "traefik.http.routers.redirs.middlewares": "redirect-to-https"
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "containous/whoami",
      "network": "BRIDGE",
      "portMappings": [
          "containerPort": 80,
          "hostPort": 0,
          "name": "http-api",
          "protocol": "tcp"
    "volumes": [
  "readinessChecks": [
      "name": "readinessCheck",
      "protocol": "HTTP",
      "path": "/",
      "portName": "http-api",
      "intervalSeconds": 30,
      "timeoutSeconds": 10,
      "httpStatusCodesForReady": [200],
      "preserveLastResponse": false
  "healthChecks": [
      "portIndex": 0,
      "protocol": "TCP",
      "gracePeriodSeconds": 30,
      "intervalSeconds": 10,
      "timeoutSeconds": 30,
      "maxConsecutiveFailures": 3
      "path": "/",
      "portIndex": 0,
      "protocol": "HTTP",
      "gracePeriodSeconds": 30,
      "intervalSeconds": 10,
      "timeoutSeconds": 30,
      "maxConsecutiveFailures": 3

This is the docker-compose.yml used to start the traefik container on localhost, that can reach marathon on the LAN through

version: '3'

    image: traefik:v2.1
    network_mode: "host"
      - "80:80"
      - "443:443"
      - "8080:8080"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./traefik.yaml:/etc/traefik/traefik.yaml"
      - "./letsencrypt:/letsencrypt"
      - "./staging/fakelerootx1.pem:/etc/ssl/certs/fakelerootx1.pem"

What did you expect to see?

I expect to be able to succeed this command:

curl -H 'Host:' -L https://localhost  --insecure

What did you see instead?

Curl response:

Gateway Timeout

traefik debug log:

Output of traefik version: (What version of Traefik are you using?)


What is your environment & configuration (arguments, toml, provider, platform, ...)?

This is my traefik.yaml

## Static configuration

  checkNewVersion: true
  sendAnonymousUsage: false

  insecureSkipVerify: false

  level: "DEBUG"

    address: ":80"
    address: ":443"

  insecure: true # enable WEB UI
  dashboard: true
  debug: true

    endpoint: ""
    watch: true
    exposedByDefault: false
    respectReadinessChecks: true
      email: ""
      storage: "/letsencrypt/acme.json"
      caServer: ""
        provider: ovh
        delayBeforeCheck: 10
      email: ""
      storage: "/letsencrypt/acme-staging.json"
      caServer: ""
        provider: ovh
        delayBeforeCheck: 10

If applicable, please paste the log output in DEBUG level (--log.level=DEBUG switch)

reverse-proxy_1  | time="2019-12-11T13:45:56Z" level=debug msg="'504 Gateway Timeout' caused by: dial tcp i/o timeout"

Note: the IP it dial as nothing to do with my LAN, it is owned by my ISP and it is not even my WAN address.

On the dashboard, IP are correctly guessed and all are reachable by telnet from traefik host.

I am out of idea, it simply does not work with marathon, any idea what is failing here?

Thanks and best!

Hi @kopax, aren't the IPs from Marathon API? And these IP could be only privates inside Marathon's infrastructure and maybe colliding with Docker's private network?

Hi @dduportal. No the IPs aren't from marathon API, but I have found what it is retrieved...

curl -L --insecure

<!DOCTYPE html><!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7" lang="fr"> <![endif]-->
<!--[if IE 7]>    <html class="no-js lt-ie9 lt-ie8" lang="fr"> <![endif]-->
<!--[if IE 8]>    <html class="no-js lt-ie9" lang="fr"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="fr"> <!--<![endif]-->
		<meta charset="utf-8" />
		<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />
		<meta name="viewport" content="width=device-width,initial-scale=1, maximum-scale=1,user-scalable=0">
		<link href="" rel="canonical">
		<link href="style.css" media="screen" rel="stylesheet" type="text/css">
		<meta name="robots" content="noindex">
		<title>Assistance Numericable</title>		
		<meta name="description" content=", le site r&amp;eacute;f&amp;eacute;rence des programmes tv, des films et s&amp;eacute;ries en vid&amp;eacute;o à la demande (VOD) et des &amp;eacute;missions de t&amp;eacute;l&amp;eacute; en Replay ! D&amp;eacute;couvrez notre Guide TV personnalisable afin de programmer votre propre soir&amp;eacute;e t&amp;eacute;l&amp;eacute;vision, selon vos go&amp;ucirc;ts, vos humeurs ou vos envies.">		
		<link rel="shortcut icon" type="image/x-icon" href="favicon.ico" />
		<link rel="stylesheet" type="text/css" href=",300,700,700italic" />
		<div class="assistance_globalContainer">
            <div id="mainContainer">
                <header class="assistance_header">
					<div class="assistance_container">
						<div class="assistance_header_left">
							<div class="assistance_header_baseline">Abonnements Internet Très Haut Débit, fibre optique et ADSL, télévision 3D HD, forfaits mobiles</div>
							<a href="/" class="assistance_header_logo">
								<img src="numericable-sfr.png" alt="Assistance Numericable">
				<div id="mainContent">        
					<div class="assistance_home">
						<div class="assistance_section assistance_sectionIntro">
							<div class="assistance_section_container">
								<div class="assistance_section_content">
									<h2>O&ugrave; trouver de l'aide ?</h2>
										<p>Votre Espace Assistance a fermé ses portes le 23 novembre 2017</p>

<p>Vous pouvez dès à présent utiliser l'Assistance SFR, disponible à l'adresse<br>

<p>Pour retrouver vos paramètres d'installation, rendez-vous sur</p>
									<a href="" class="assistance_popinLink assistance_button assistance_button-green assistance_button-home">Acc&eacute;der &agrave;</a>&nbsp;&nbsp;<a href="" class="assistance_popinLink assistance_button assistance_button-green assistance_button-home">Acc&eacute;der &agrave;</a>&nbsp;&nbsp;<a href="" class="assistance_popinLink assistance_button assistance_button-green assistance_button-home">Contactez-nous</a>
						<div class="assistance_homeFooter">
							<div class="assistance_homeFooter_container assistance_homeFooter_content">
								&copy; 2017 Numericable. Tous droits r&eacute;serv&eacute;s.


This is the screenshot of that page:

My ISP use that page to indicate that an unknown DNS was requested.

marathon is running an app and show an ip with the host where the app run, for example, for me marathon show, this dns is resolvable (normally), I am pretty sure that traefik can't resolve it and return a fallback IP from my ISP.

I am 99% sure the issue is that, if I tell marathon not to use the host, I can prove that using the IP instead of the DNS will work.

The issue is that traefic reverse proxy try to redirect to that DNS so traefik MUST be able to resolve it

It can't resolve the DNS even if the host as no network misconfiguration. if it was an IP, it could probably serve the page.

To me traefik is having a big bug, either traefik skip some DNS resolution, or perform some DNS resolution

What does traefik to solve DNS? Any idea what I can do?

My ISP host a page in case a DNS is not known. This is the IP that traefik return to me, while it should return another IP (the one from marathon). I have never configured that one anywhere, it's not mine.

I don't get why traefic is getting crazy about it, any idea what to do?


This is the DNS returning wrong page:

I did some tcpdump,

My /etc/resolv.conf:

# Generated by NetworkManager

It seems that is really redirecting to

I was configuring marathon with an intranet host (dev-11), this was the tcp dump:

/ # tcpdump -ni any port 53
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes

23:37:25.891422 IP > 378+ AAAA? (39)
23:37:25.891479 IP > 43249+ A? (39)
23:37:25.904896 IP > 378 1/1/0 CNAME (120)
23:37:25.930865 IP > 43249 2/0/0 CNAME, A (84)
23:37:26.739694 IP > 4284+ A? (38)
23:37:26.801338 IP > 4284 3/0/0 CNAME, CNAME, A (138)
23:37:41.825827 IP > 11463+ A? (38)
23:37:41.838653 IP > 11463 3/0/0 CNAME, CNAME, A (138)
23:37:41.865510 IP > 5711+ A? (31)
23:37:41.880401 IP > 5711 1/0/0 A (47)
23:37:41.903823 IP > 58412+ A? (43)
23:37:41.925646 IP > 58412 2/0/0 CNAME, A (81)
23:37:41.947066 IP > 42171+ A? (38)
23:37:41.961610 IP > 42171 1/0/0 A (54)
23:37:50.180251 IP > 14439+ A? (47)
23:37:50.191554 IP > 14439 4/0/0 A, A, A, A (111)

It was fallbacking to, so I have configured a fqdn ( with marathon, now the tcpdump look like:

 # tcpdump -ni any port 53
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
23:45:51.408693 IP > 60928+ A? (38)
23:45:51.435049 IP > 60928 3/0/0 CNAME, CNAME, A (138)
23:45:51.457513 IP > 13877+ A? (32)
23:45:51.465513 IP > 13877 3/0/0 CNAME, CNAME, A (110)
23:45:51.488500 IP > 41661+ A? (38)
23:45:51.497684 IP > 41661 1/0/0 A (54)
23:45:56.754038 IP > 6818+ AAAA? (35)
23:45:56.754055 IP > 59618+ A? (35)
23:45:57.177981 IP > 6818 NXDomain 0/1/0 (91)
23:45:57.178003 IP > 59618 NXDomain 0/1/0 (91)
23:45:57.178478 IP > 17534+ AAAA? (50)
23:45:57.178504 IP > 63898+ A? (50)
23:45:57.205267 IP > 63898 2/0/0 CNAME, A (95)
23:45:57.217886 IP > 17534 1/1/0 CNAME (131)
23:46:06.521472 IP > 33403+ A? (38)
23:46:06.552800 IP > 33403 3/0/0 CNAME, CNAME, A (138)
23:46:06.574352 IP > 24754+ A? (51)
23:46:06.597444 IP > 24754 6/0/0 CNAME, CNAME, CNAME, CNAME, CNAME, A (225)
23:46:06.618327 IP > 51941+ A? (36)
23:46:06.634170 IP > 51941 1/0/0 A (52)

I am still confused, because my /etc/hosts is having :

/ # cat /etc/hosts | grep dev-11 dev-11

host file seems to be ignored by traefik...

After doing:

  • using marathon with FQDN
  • creating a real DNS,
  • opening marathon private port publicly

I was able to query my server. But I don't know how to fix it, because the port can't remain publicly open, and I don't know why /etc/hosts does not get resolved causing all those issue.

Any idea?

Not related to #1243, it seems that /etc/hosts is ignored totally by traeffic.

Related to

/etc/nsswitch.conf is absent from the system, doing:

echo "hosts: files dns" > /etc/nsswitch.conf

will solve the issue of /etc/hosts being ignore.

Have you try this option?

Yes I did.

I submited a fix but it was rejected because it was not on the official Dockerfile. I don't know where is the official Dockerfile.

Status: you opened