Consolidated 404 issue thread in versions since 2.2.1

It seems that there are quite a few people are getting intermittent 404 since they upgrade from 2.2.1 to a later version, and when they roll back their configuration changes to 2.2.1 the issue goes away.

Examples:

https://community.containo.us/t/v2-https-domain-and-subdomain-error-404/6848
https://community.containo.us/t/v2-intermittent-404-errors-across-our-docker-containers/6853
https://community.containo.us/t/404-even-when-route-exists/6854
https://community.containo.us/t/k8s-traefik-404-not-found-all-routes-logs-show-skipping-kubernetes-event-kind/6845

First of all, generally 404 means that route could not be matched, so if you have 404 it does not automatically means that you have this issue.

If you have intermittent 404, which is fixed by downgrading to 2.2.1 without changing the configuration, then you might be affected. If not, I ask you to kindly post to a different thread.

At this moment it is not clear if this is a bug, or a subtle configuration interpreting change that leads to previously working configuration to misbehave.

Since this is an intermittent issue it is very difficult to come up with a small reproduction case, that we can give to the traefik developers, so that could try and figure out what is the cause of this issue.

In this thread I'd like to ask you to help me to come up with this reproduction case. I'll start with saying that I myself have not experienced this issue so far.

Please post your configuration, as minimal as possible, that reproduces the issue. That would help to locate it and if this is a bug to, eventually, fix it.

  • It appears that this happens with or without swarm. Please post your configuration without swarm to simplify the situation. If you can only reproduce with swarm, than post it anyway, of course.
  • if you can reproduce it without SSL it also preferable to setup with SSL since it's simpler
  • post all docker-compose files and all other toml/yaml files involved in configuration
  • with intermittent issues this is difficult, but please try to describe how the issue could be demonstrated

The goal is that I (and others) can take your configs, run them locally and see the issue you are experiencing. Thank you for your help!

Here's my contribution. I hope this helps

Traefik, Whoami and Portainer "docker-compose.yml"

version: "3.8"

services:
  traefik:
    image: "traefik:latest"
    restart: unless-stopped
    container_name: traefik
    ports:
      - "80:80"
      - "443:443"
      - "53:53"
      - "53:53/udp"
      - "3306:3306"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "$PWD/traefik/:/etc/traefik/"
      - "$PWD/certs/traefik:/certs/"
      - "$PWD/logs/traefik:/logs"
      - /etc/localtime:/etc/localtime:ro
    networks:
      - web
      - backend
      - internal
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.api.rule=Host(`monitor.domain.lan`)"
      - "traefik.http.routers.api.service=api@internal"
      - "traefik.http.routers.api.tls=true"
      - "traefik.http.routers.api.middlewares=auth"
      - "traefik.http.middlewares.auth.basicauth.users=admin:password"

  portainer:
    image: portainer/portainer
    restart: unless-stopped
    container_name: portainer
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - $PWD/../data/portainer:/data
    networks:
      - backend
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=backend"
      ## HTTP Routers
      - "traefik.http.routers.portainer-rtr.entrypoints=websec-ep"
      - "traefik.http.routers.portainer-rtr.tls=true"
      - "traefik.http.routers.portainer-rtr.rule=Host(`portainer.domain.lan`) || PathPrefix(`/portainer`) || Host(`portainer.lan`) "
      ## Middleware
      - "traefik.http.routers.portainer-rtr.middlewares=portainer-stripprefix"
      - "traefik.http.middlewares.portainer-stripprefix.stripprefix.prefixes=/portainer"
      ## HTTP Services
      - "traefik.http.routers.portainer-rtr.service=portainer-svc"
      - "traefik.http.services.portainer-svc.loadbalancer.server.port=9000"

  whoami:
    # A container that exposes an API to show its IP address
    image: containous/whoami
    restart: unless-stopped
    container_name: whoami
    networks:
      - backend
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=backend"
      ## HTTP Routers
      - "traefik.http.routers.whoami-rtr.entrypoints=websec-ep"
      - "traefik.http.routers.whoami-rtr.tls=true"
      - "traefik.http.routers.whoami-rtr.rule=Host(`whoami.domain.lan`) || Path(`/whoami`)"
      ## HTTP Services
      - "traefik.http.routers.whoami-rtr.service=whoami-svc"
      - "traefik.http.services.whoami-svc.loadbalancer.server.port=80"

networks:
  web:
   name: "web"
  backend:
   name: "backend"
  internal:
   name: "internal"

Traefik Yaml config file

global:
  checkNewVersion: true
  sendAnonymousUsage: false

log:
  level: debug
  filePath: /logs/logs.txt

api:
  dashboard: true
#  insecure: true

entryPoints:
  web-ep:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websec-ep
          scheme: https
          permanent: true

  websec-ep:
    address: ":443"
#  prometheus-ep:
#    address: ":9090"
  dns-ep:
    address: ":53"
  dns-udp-ep:
    address: ":53/udp"
  db-ep:
    address: ":3306"

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
  file:
    directory: /etc/traefik/
    watch: true

metrics:
  prometheus: {}
  prometheus:
    addEntryPointsLabels: true
    addServicesLabels: true
    entryPoint: web-ep

tls.yml file

tls:
  certificates:
    - certFile: /certs/domain.lan.crt
      keyFile: /certs/domain.lan.key
    - certFile: /certs/portainer.lan.crt
      keyFile: /certs/portainer.lan.key

Other apps

version: '3.8'
#networks:
#  backend:
#    driver: bridge

networks:
  backend:
    external: true

#volumes:
#    prometheus_data: {}
#    grafana_data: {}

services:
  prometheus:
    image: prom/prometheus:latest
    user: root
    restart: unless-stopped
    container_name: prometheus
    volumes:
#      - $PWD/../conf/prometheus:/etc/prometheus/
#      - $PWD/../conf/prometheus/prometheus.yml:/etc/prometheus/conf/prometheus.yml:ro
      - $PWD/../conf/prometheus/prometheus.yml:/etc/prometheus/conf/prometheus.yml

    command:
      - '--config.file=/etc/prometheus/conf/prometheus.yml'
      - '--storage.tsdb.path=data'
      - '--web.console.libraries=console_libraries'
      - '--web.console.templates=consoles'
      - '--storage.tsdb.retention.time=7d'
      - '--web.enable-lifecycle'
    networks:
      - backend
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=backend"
      ## HTTP Routers
      - "traefik.http.routers.prometheus-rtr.entrypoints=websec-ep"
      - "traefik.http.routers.prometheus-rtr.tls=true"
      - "traefik.http.routers.prometheus-rtr.rule=Host(`prometheus.domain.lan`) || Path(`/prometheus`)"
      ## HTTP Services
      - "traefik.http.routers.prometheus-rtr.service=prometheus-svc"
      - "traefik.http.services.prometheus-svc.loadbalancer.server.port=9090"

  grafana:
    image: grafana/grafana:latest
    user: root
    restart: unless-stopped
    container_name: grafana
    volumes:
#      - grafana_data:/var/lib/grafana
       - $PWD/../data/grafana:/data
       - $PWD/../conf/grafana:/etc/grafana
       - $PWD/../logs/grafana:/var/log
    networks:
      - backend
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=backend"
      ## HTTP Routers
      - "traefik.http.routers.grafana-rtr.entrypoints=websec-ep"
      - "traefik.http.routers.grafana-rtr.tls=true"
      - "traefik.http.routers.grafana-rtr.rule=Host(`grafana.domain.lan`) || Path(`/grafana`)"
      ## HTTP Services
      - "traefik.http.routers.grafana-rtr.service=grafana-svc"
      - "traefik.http.services.grafana-svc.loadbalancer.server.port=3000"

Pihole & Adminer

version: '3.8'

networks:
  backend:
    external: true

services:
  pihole:
    image: pihole/pihole:latest
#    user: root
    restart: unless-stopped
    container_name: pihole
    volumes:
      - "$PWD/../data/pihole/etc-pihole:/etc/pihole/"
      - "$PWD/../data/pihole/etc-dnsmasq:/etc/dnsmasq.d/"
#      - "$PWD/../logs/pihole/:/var/log"
    networks:
      - backend
    environment:
      WEBPASSWORD: 'pihole'
      DNS1: 1.1.1.1
      DNS2: 1.0.0.1
      TZ: Europe/London
    dns:
      - 127.0.0.1
      - 1.1.1.1
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=backend"

      ## TCP Routers/Service
      - "traefik.tcp.routers.pihole-rtr.entrypoints=dns-ep"
      - "traefik.tcp.routers.pihole-rtr.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.pihole-rtr.service=pihole-svc"
      - "traefik.tcp.services.pihole-svc.loadbalancer.server.port=53"

      ## UDP Routers/Services
      - "traefik.udp.routers.pihole-udp-rtr.entrypoints=dns-udp-ep"
      - "traefik.udp.routers.pihole-udp-rtr.service=pihole-udp-svc"
      - "traefik.udp.services.pihole-udp-svc.loadbalancer.server.port=53"

      ## HTTP Routers/Service
      - "traefik.http.routers.pihole-admin-rtr.entrypoints=websec-ep"
      - "traefik.http.routers.pihole-admin-rtr.tls=true"
      - "traefik.http.routers.pihole-admin-rtr.rule=Host(`pihole.domain.lan`)"
      - "traefik.http.routers.pihole-admin-rtr.service=pihole-admin-svc"
      - "traefik.http.services.pihole-admin-svc.loadbalancer.server.port=80"

  adminer:
    image: adminer:latest
    restart: unless-stopped
    container_name: adminer
    networks:
      - backend
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=backend"
      ## HTTP Routers
      - "traefik.http.routers.adminer-rtr.entrypoints=websec-ep"
      - "traefik.http.routers.adminer-rtr.tls=true"
      - "traefik.http.routers.adminer-rtr.rule=Host(`adminer.domain.lan`) || Path(`/adminer`)"
      ## HTTP Services
      - "traefik.http.routers.adminer-rtr.service=adminer-svc"
      - "traefik.http.services.adminer-svc.loadbalancer.server.port=8080"

Grafana ini file

##################### Grafana Configuration Defaults #####################
#
# Do not modify this file in grafana installs
#

# possible values : production, development
app_mode = production

# instance name, defaults to HOSTNAME environment variable value or hostname if HOSTNAME var is empty
instance_name = ${HOSTNAME}

#################################### Paths ###############################
[paths]
# Path to where grafana can store temp files, sessions, and the sqlite3 db (if that is used)
data = /data

# Temporary files in `data` directory older than given duration will be removed
temp_data_lifetime = 24h

# Directory where grafana can store logs
#logs = data/log
logs = /var/log

# Directory where grafana will automatically scan and look for plugins
plugins = data/plugins

# folder that contains provisioning config files that grafana will apply on startup and while running.
provisioning = conf/provisioning

#################################### Server ##############################
[server]
# Protocol (http, https, h2, socket)
protocol = http

# The ip address to bind to, empty will bind to all interfaces
http_addr =

# The http port to use
http_port = 3000

# The public facing domain name used to access grafana from a browser
domain = localhost

# Redirect to correct domain if host header does not match domain
# Prevents DNS rebinding attacks
enforce_domain = false

# The full public facing url
root_url = %(protocol)s://%(domain)s:%(http_port)s/

# Serve Grafana from subpath specified in `root_url` setting. By default it is set to `false` for compatibility reasons.
serve_from_sub_path = false

# Log web requests
router_logging = false

# the path relative working path
static_root_path = public

# enable gzip
enable_gzip = false

# https certs & key file
cert_file =
cert_key =

# Unix socket path
socket = /tmp/grafana.sock

#################################### Database ############################
[database]
# You can configure the database connection by specifying type, host, name, user and password
# as separate properties or as on string using the url property.

# Either "mysql", "postgres" or "sqlite3", it's your choice
type = sqlite3
host = 127.0.0.1:3306
name = grafana
user = root
# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;"""
password =
# Use either URL or the previous fields to configure the database
# Example: mysql://user:secret@host:port/database
url =

# Max idle conn setting default is 2
max_idle_conn = 2

# Max conn setting default is 0 (mean not set)
max_open_conn =

# Connection Max Lifetime default is 14400 (means 14400 seconds or 4 hours)
conn_max_lifetime = 14400

# Set to true to log the sql calls and execution times.
log_queries =

# For "postgres", use either "disable", "require" or "verify-full"
# For "mysql", use either "true", "false", or "skip-verify".
ssl_mode = disable

ca_cert_path =
client_key_path =
client_cert_path =
server_cert_name =

# For "sqlite3" only, path relative to data_path setting
path = grafana.db

# For "sqlite3" only. cache mode setting used for connecting to the database
cache_mode = private

#################################### Cache server #############################
[remote_cache]
# Either "redis", "memcached" or "database" default is "database"
type = database

# cache connectionstring options
# database: will use Grafana primary database.
# redis: config like redis server e.g. `addr=127.0.0.1:6379,pool_size=100,db=0,ssl=false`. Only addr is required. ssl may be 'true', 'false', or 'insecure'.
# memcache: 127.0.0.1:11211
connstr =

#################################### Data proxy ###########################
[dataproxy]

# This enables data proxy logging, default is false
logging = false

# How long the data proxy waits before timing out, default is 30 seconds.
# This setting also applies to core backend HTTP data sources where query requests use an HTTP client with timeout set.
timeout = 30

# If enabled and user is not anonymous, data proxy will add X-Grafana-User header with username into the request, default is false.
send_user_header = false

#################################### Analytics ###########################
[analytics]
# Server reporting, sends usage counters to stats.grafana.org every 24 hours.
# No ip addresses are being tracked, only simple counters to track
# running instances, dashboard and error counts. It is very helpful to us.
# Change this option to false to disable reporting.
reporting_enabled = false

# Set to false to disable all checks to https://grafana.com
# for new versions (grafana itself and plugins), check is used
# in some UI views to notify that grafana or plugin update exists
# This option does not cause any auto updates, nor send any information
# only a GET request to https://grafana.com to get latest versions
check_for_updates = true

# Google Analytics universal tracking code, only enabled if you specify an id here
google_analytics_ua_id =

# Google Tag Manager ID, only enabled if you specify an id here
google_tag_manager_id =

#################################### Security ############################
[security]
# disable creation of admin user on first start of grafana
disable_initial_admin_creation = false

# default admin user, created on startup
admin_user = admin

# default admin password, can be changed before first start of grafana, or in profile settings
admin_password = admin

# used for signing
secret_key = SW2YcwTIb9zpOOhoPsMm

# disable gravatar profile images
disable_gravatar = false

# data source proxy whitelist (ip_or_domain:port separated by spaces)
data_source_proxy_whitelist =

# disable protection against brute force login attempts
disable_brute_force_login_protection = false

# set to true if you host Grafana behind HTTPS. default is false.
cookie_secure = false

# set cookie SameSite attribute. defaults to `lax`. can be set to "lax", "strict", "none" and "disabled"
cookie_samesite = lax

# set to true if you want to allow browsers to render Grafana in a <frame>, <iframe>, <embed> or <object>. default is false.
allow_embedding = false

# Set to true if you want to enable http strict transport security (HSTS) response header.
# This is only sent when HTTPS is enabled in this configuration.
# HSTS tells browsers that the site should only be accessed using HTTPS.
strict_transport_security = false

# Sets how long a browser should cache HSTS. Only applied if strict_transport_security is enabled.
strict_transport_security_max_age_seconds = 86400

# Set to true if to enable HSTS preloading option. Only applied if strict_transport_security is enabled.
strict_transport_security_preload = false

# Set to true if to enable the HSTS includeSubDomains option. Only applied if strict_transport_security is enabled.
strict_transport_security_subdomains = false

# Set to true to enable the X-Content-Type-Options response header.
# The X-Content-Type-Options response HTTP header is a marker used by the server to indicate that the MIME types advertised
# in the Content-Type headers should not be changed and be followed.
x_content_type_options = true

# Set to true to enable the X-XSS-Protection header, which tells browsers to stop pages from loading
# when they detect reflected cross-site scripting (XSS) attacks.
x_xss_protection = true


#################################### Snapshots ###########################
[snapshots]
# snapshot sharing options
external_enabled = true
external_snapshot_url = https://snapshots-origin.raintank.io
external_snapshot_name = Publish to snapshot.raintank.io

# Set to true to enable this Grafana instance act as an external snapshot server and allow unauthenticated requests for
# creating and deleting snapshots.
public_mode = false

# remove expired snapshot
snapshot_remove_expired = true

#################################### Dashboards ##################

[dashboards]
# Number dashboard versions to keep (per dashboard). Default: 20, Minimum: 1
versions_to_keep = 20

# Minimum dashboard refresh interval. When set, this will restrict users to set the refresh interval of a dashboard lower than given interval. Per default this is 5 seconds.
# The interval string is a possibly signed sequence of decimal numbers, followed by a unit suffix (ms, s, m, h, d), e.g. 30s or 1m.
min_refresh_interval = 5s

# Path to the default home dashboard. If this value is empty, then Grafana uses StaticRootPath + "dashboards/home.json"
default_home_dashboard_path =

#################################### Users ###############################
[users]
# disable user signup / registration
allow_sign_up = false

# Allow non admin users to create organizations
allow_org_create = false

# Set to true to automatically assign new users to the default organization (id 1)
auto_assign_org = true

# Set this value to automatically add new users to the provided organization (if auto_assign_org above is set to true)
auto_assign_org_id = 1

# Default role new users will be automatically assigned (if auto_assign_org above is set to true)
auto_assign_org_role = Viewer

# Require email validation before sign up completes
verify_email_enabled = false

# Background text for the user field on the login page
login_hint = email or username
password_hint = password

# Default UI theme ("dark" or "light")
default_theme = dark

# External user management
external_manage_link_url =
external_manage_link_name =
external_manage_info =

# Viewers can edit/inspect dashboard settings in the browser. But not save the dashboard.
viewers_can_edit = false

# Editors can administrate dashboard, folders and teams they create
editors_can_admin = false

[auth]
# Login cookie name
login_cookie_name = grafana_session

# The lifetime (days) an authenticated user can be inactive before being required to login at next visit. Default is 7 days.
login_maximum_inactive_lifetime_days = 7

# The maximum lifetime (days) an authenticated user can be logged in since login time before being required to login. Default is 30 days.
login_maximum_lifetime_days = 30

# How often should auth tokens be rotated for authenticated users when being active. The default is each 10 minutes.
token_rotation_interval_minutes = 10

# Set to true to disable (hide) the login form, useful if you use OAuth
disable_login_form = false

# Set to true to disable the signout link in the side menu. useful if you use auth.proxy
disable_signout_menu = false

# URL to redirect the user to after sign out
signout_redirect_url =

# Set to true to attempt login with OAuth automatically, skipping the login screen.
# This setting is ignored if multiple OAuth providers are configured.
oauth_auto_login = false

# OAuth state max age cookie duration. Defaults to 60 seconds.
oauth_state_cookie_max_age = 60

# limit of api_key seconds to live before expiration
api_key_max_seconds_to_live = -1

#################################### Anonymous Auth ######################
[auth.anonymous]
# enable anonymous access
enabled = false

# specify organization name that should be used for unauthenticated users
org_name = Main Org.

# specify role for unauthenticated users
org_role = Viewer

# mask the Grafana version number for unauthenticated users
hide_version = false

#################################### Github Auth #########################
[auth.github]
enabled = false
allow_sign_up = true
client_id = some_id
client_secret = some_secret
scopes = user:email,read:org
auth_url = https://github.com/login/oauth/authorize
token_url = https://github.com/login/oauth/access_token
api_url = https://api.github.com/user
allowed_domains =
team_ids =
allowed_organizations =

#################################### GitLab Auth #########################
[auth.gitlab]
enabled = false
allow_sign_up = true
client_id = some_id
client_secret = some_secret
scopes = api
auth_url = https://gitlab.com/oauth/authorize
token_url = https://gitlab.com/oauth/token
api_url = https://gitlab.com/api/v4
allowed_domains =
allowed_groups =

#################################### Google Auth #########################
[auth.google]
enabled = false
allow_sign_up = true
client_id = some_client_id
client_secret = some_client_secret
scopes = https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/userinfo.email
auth_url = https://accounts.google.com/o/oauth2/auth
token_url = https://accounts.google.com/o/oauth2/token
api_url = https://www.googleapis.com/oauth2/v1/userinfo
allowed_domains =
hosted_domain =

#################################### Grafana.com Auth ####################
# legacy key names (so they work in env variables)
[auth.grafananet]
enabled = false
allow_sign_up = true
client_id = some_id
client_secret = some_secret
scopes = user:email
allowed_organizations =

[auth.grafana_com]
enabled = false
allow_sign_up = true
client_id = some_id
client_secret = some_secret
scopes = user:email
allowed_organizations =

#################################### Generic OAuth #######################
[auth.generic_oauth]
name = OAuth
enabled = false
allow_sign_up = true
client_id = some_id
client_secret = some_secret
scopes = user:email
email_attribute_name = email:primary
email_attribute_path =
role_attribute_path =
auth_url =
token_url =
api_url =
allowed_domains =
team_ids =
allowed_organizations =
tls_skip_verify_insecure = false
tls_client_cert =
tls_client_key =
tls_client_ca =

#################################### Basic Auth ##########################
[auth.basic]
enabled = true

#################################### Auth Proxy ##########################
[auth.proxy]
enabled = false
header_name = X-WEBAUTH-USER
header_property = username
auto_sign_up = true
# Deprecated, use sync_ttl instead
ldap_sync_ttl = 60
sync_ttl = 60
whitelist =
headers =
enable_login_token = false

#################################### Logging ##########################
[log]
# Either "console", "file", "syslog". Default is console and file
# Use space to separate multiple modes, e.g. "console file"
mode = console file

# Either "debug", "info", "warn", "error", "critical", default is "info"
level = info

# optional settings to set different levels for specific loggers. Ex filters = sqlstore:debug
filters =

# For "console" mode only
[log.console]
level =

# log line format, valid options are text, console and json
format = console

# For "file" mode only
[log.file]
level =

# log line format, valid options are text, console and json
format = text

# This enables automated log rotate(switch of following options), default is true
log_rotate = true

# Max line number of single file, default is 1000000
max_lines = 1000000

# Max size shift of single file, default is 28 means 1 << 28, 256MB
max_size_shift = 28

# Segment log daily, default is true
daily_rotate = true

# Expired days of log file(delete after max days), default is 7
max_days = 7

[log.syslog]
level =

# log line format, valid options are text, console and json
format = text

# Syslog network type and address. This can be udp, tcp, or unix. If left blank, the default unix endpoints will be used.
network =
address =

# Syslog facility. user, daemon and local0 through local7 are valid.
facility =

# Syslog tag. By default, the process' argv[0] is used.
tag =

#################################### Explore #############################
[explore]
# Enable the Explore section
enabled = true

#################################### Internal Grafana Metrics ############
# Metrics available at HTTP API Url /metrics
[metrics]
enabled              = true
interval_seconds     = 10
# Disable total stats (stat_totals_*) metrics to be generated
disable_total_stats = false

#If both are set, basic auth will be required for the metrics endpoint.
basic_auth_username =
basic_auth_password =

# Send internal Grafana metrics to graphite
[metrics.graphite]
# Enable by setting the address setting (ex localhost:2003)
address =
prefix = prod.grafana.%(instance_name)s.

#################################### Grafana.com integration  ##########################
[grafana_net]
url = https://grafana.com

[grafana_com]
url = https://grafana.com

#################################### Distributed tracing ############
[tracing.jaeger]
# jaeger destination (ex localhost:6831)
address =
# tag that will always be included in when creating new spans. ex (tag1:value1,tag2:value2)
always_included_tag =
# Type specifies the type of the sampler: const, probabilistic, rateLimiting, or remote
sampler_type = const
# jaeger samplerconfig param
# for "const" sampler, 0 or 1 for always false/true respectively
# for "probabilistic" sampler, a probability between 0 and 1
# for "rateLimiting" sampler, the number of spans per second
# for "remote" sampler, param is the same as for "probabilistic"
# and indicates the initial sampling rate before the actual one
# is received from the mothership
sampler_param = 1
# Whether or not to use Zipkin span propagation (x-b3- HTTP headers).
zipkin_propagation = false
# Setting this to true disables shared RPC spans.
# Not disabling is the most common setting when using Zipkin elsewhere in your infrastructure.
disable_shared_zipkin_spans = false

#################################### External Image Storage ##############
[external_image_storage]
# Used for uploading images to public servers so they can be included in slack/email messages.
# You can choose between (s3, webdav, gcs, azure_blob, local)
provider =

[external_image_storage.s3]
endpoint =
path_style_access =
bucket_url =
bucket =
region =
path =
access_key =
secret_key =

[external_image_storage.webdav]
url =
username =
password =
public_url =

[external_image_storage.gcs]
key_file =
bucket =
path =

[external_image_storage.azure_blob]
account_name =
account_key =
container_name =

[external_image_storage.local]
# does not require any configuration

[rendering]
# Options to configure a remote HTTP image rendering service, e.g. using https://github.com/grafana/grafana-image-renderer.
# URL to a remote HTTP image renderer service, e.g. http://localhost:8081/render, will enable Grafana to render panels and dashboards to PNG-images using HTTP requests to an external service.
server_url =
# If the remote HTTP image renderer service runs on a different server than the Grafana server you may have to configure this to a URL where Grafana is reachable, e.g. http://grafana.domain/.
callback_url =
# Concurrent render request limit affects when the /render HTTP endpoint is used. Rendering many images at the same time can overload the server,
# which this setting can help protect against by only allowing a certain amount of concurrent requests.
concurrent_render_request_limit = 30

[panels]
# here for to support old env variables, can remove after a few months
enable_alpha = false
disable_sanitize_html = false

[plugins]
enable_alpha = false
app_tls_skip_verify_insecure = false
# Enter a comma-separated list of plugin identifiers to identify plugins that are allowed to be loaded even if they lack a valid signature.
allow_loading_unsigned_plugins =

#################################### Grafana Image Renderer Plugin ##########################
[plugin.grafana-image-renderer]
# Instruct headless browser instance to use a default timezone when not provided by Grafana, e.g. when rendering panel image of alert.
# See ICU’s metaZones.txt (https://cs.chromium.org/chromium/src/third_party/icu/source/data/misc/metaZones.txt) for a list of supported
# timezone IDs. Fallbacks to TZ environment variable if not set.
rendering_timezone =

# Instruct headless browser instance to use a default language when not provided by Grafana, e.g. when rendering panel image of alert.
# Please refer to the HTTP header Accept-Language to understand how to format this value, e.g. 'fr-CH, fr;q=0.9, en;q=0.8, de;q=0.7, *;q=0.5'.
rendering_language =

# Instruct headless browser instance to use a default device scale factor when not provided by Grafana, e.g. when rendering panel image of alert.
# Default is 1. Using a higher value will produce more detailed images (higher DPI), but will require more disk space to store an image.
rendering_viewport_device_scale_factor =

# Instruct headless browser instance whether to ignore HTTPS errors during navigation. Per default HTTPS errors are not ignored. Due to
# the security risk it's not recommended to ignore HTTPS errors.
rendering_ignore_https_errors =

# Instruct headless browser instance whether to capture and log verbose information when rendering an image. Default is false and will
# only capture and log error messages. When enabled, debug messages are captured and logged as well.
# For the verbose information to be included in the Grafana server log you have to adjust the rendering log level to debug, configure
# [log].filter = rendering:debug.
rendering_verbose_logging =

# Instruct headless browser instance whether to output its debug and error messages into running process of remote rendering service.
# Default is false. This can be useful to enable (true) when troubleshooting.
rendering_dumpio =

# Additional arguments to pass to the headless browser instance. Default is --no-sandbox. The list of Chromium flags can be found
# here (https://peter.sh/experiments/chromium-command-line-switches/). Multiple arguments is separated with comma-character.
rendering_args =

# You can configure the plugin to use a different browser binary instead of the pre-packaged version of Chromium.
# Please note that this is not recommended, since you may encounter problems if the installed version of Chrome/Chromium is not
# compatible with the plugin.
rendering_chrome_bin =

# Instruct how headless browser instances are created. Default is 'default' and will create a new browser instance on each request.
# Mode 'clustered' will make sure that only a maximum of browsers/incognito pages can execute concurrently.
# Mode 'reusable' will have one browser instance and will create a new incognito page on each request.
rendering_mode =

# When rendering_mode = clustered you can instruct how many browsers or incognito pages can execute concurrently. Default is 'browser'
# and will cluster using browser instances.
# Mode 'context' will cluster using incognito pages.
rendering_clustering_mode =
# When rendering_mode = clustered you can define maximum number of browser instances/incognito pages that can execute concurrently..
rendering_clustering_max_concurrency =

# Limit the maximum viewport width, height and device scale factor that can be requested.
rendering_viewport_max_width =
rendering_viewport_max_height =
rendering_viewport_max_device_scale_factor =

# Change the listening host and port of the gRPC server. Default host is 127.0.0.1 and default port is 0 and will automatically assign
# a port not in use.
grpc_host =
grpc_port =

[enterprise]
license_path =

[feature_toggles]
# enable features, separated by spaces
enable =

Prometheus file

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090','traefik:80']

@tam481 thank you for this. This does not seem to be a complete configuration, as prometheus and grafana configuration are missing. Without prometheus and grafana I was able to get it up and running with minimal changes (removed the auth middleware and I do not have the certs) and I cannot reproduce the issue - have not seen a single 404 everything seems to be running as intended.

Ludovic hints that in 2.2.2 and 2.2.3 it could be because of domain fronting detection that was enabled by default. It should be disabled by default in 2.2.4, so all who experience this issue are encouraged to try 2.2.4 and report if it's working or not.

This github comment deals with underlying reasons for this issue: https://github.com/containous/traefik/issues/7020#issuecomment-656753476

I experienced the issue with 2.2.4 as I jumped from 2.2.1 to that release which I believe at the time was only 4 hours old :slight_smile:

I don't have any fancy config in my influxdb and grafana config files but I'll update my reply to include these too.

My set up is actually a lab hence the domain is domain.lan as that is my test domain. Also, the certs are self generated and signed by my own CA which is trusted on my lab machine. Again for testing.

I can confirm that adding

 global:
   insecureSNI: true

To the config file fixed the issue for me in 2.2.4

I've just started reading about this domain fronting stuff as I've never heard of it before.

I've also noticed new entries in the log file in debug mode

level=debug msg="Serving default certificate for request: \"traefik\""

I don't know what that means or what it is using it for yet

I'm also seeing a lot of entries like these

time="2020-07-12T02:44:20+01:00" level=debug msg="Router reached with Host(\"pihole.domain.lan\") different from SNI(\"monitor.domain.lan\")"
time="2020-07-12T02:44:20+01:00" level=debug msg="Router reached with Host(\"grafana.domain.lan\") different from SNI(\"monitor.domain.lan\")"
time="2020-07-12T02:44:20+01:00" level=debug msg="Router reached with Host(\"grafana.domain.lan\") different from SNI(\"monitor.domain.lan\")"
time="2020-07-12T02:44:21+01:00" level=debug msg="Router reached with Host(\"grafana.domain.lan\") different from SNI(\"monitor.domain.lan\")"
time="2020-07-12T02:44:21+01:00" level=debug msg="Router reached with Host(\"grafana.domain.lan\") different from SNI(\"monitor.domain.lan\")"
time="2020-07-12T02:44:21+01:00" level=debug msg="Router reached with Host(\"grafana.domain.lan\") different from SNI(\"monitor.domain.lan\")"
time="2020-07-12T02:44:22+01:00" level=debug msg="Router reached with Host(\"grafana.domain.lan\") different from SNI(\"monitor.domain.lan\")"
time="2020-07-12T02:44:22+01:00" level=debug msg="Router reached with Host(\"pihole.domain.lan\") different from SNI(\"monitor.domain.lan\")"
time="2020-07-12T02:44:24+01:00" level=debug msg="Router reached with Host(\"pihole.domain.lan\") different from SNI(\"monitor.domain.lan\")"
time="2020-07-12T02:44:24+01:00" level=debug msg="Router reached with Host(\"portainer.domain.lan\") different from SNI(\"monitor.domain.lan\")"
time="2020-07-12T02:44:24+01:00" level=debug msg="Router reached with Host(\"portainer.domain.lan\") different from SNI(\"monitor.domain.lan\")"
time="2020-07-12T02:44:24+01:00" level=debug msg="Router reached with Host(\"portainer.domain.lan\") different from SNI(\"monitor.domain.lan\")"
time="2020-07-12T02:44:24+01:00" level=debug msg="Router reached with Host(\"portainer.domain.lan\") different from SNI(\"monitor.domain.lan\")"
time="2020-07-12T02:44:25+01:00" level=debug msg="Router reached with Host(\"portainer.domain.lan\") different from SNI(\"monitor.domain.lan\")"
time="2020-07-12T02:44:25+01:00" level=debug msg="Router reached with Host(\"portainer.domain.lan\") different from SNI(\"monitor.domain.lan\")"
time="2020-07-12T02:44:25+01:00" level=debug msg="Router reached with Host(\"portainer.domain.lan\") different from SNI(\"monitor.domain.lan\")"
time="2020-07-12T02:44:25+01:00" level=debug msg="Router reached with Host(\"portainer.domain.lan\") different from SNI(\"monitor.domain.lan\")"
time="2020-07-12T02:44:25+01:00" level=debug msg="Router reached with Host(\"portainer.domain.lan\") different from SNI(\"monitor.domain.lan\")"
time="2020-07-12T02:44:25+01:00" level=debug msg="Router reached with Host(\"portainer.domain.lan\") different from SNI(\"monitor.domain.lan\")"
time="2020-07-12T02:44:26+01:00" level=debug msg="Router reached with Host(\"pihole.domain.lan\") different from SNI(\"monitor.domain.lan\")"
time="2020-07-12T02:44:28+01:00" level=debug msg="Router reached with Host(\"pihole.domain.lan\") different from SNI(\"monitor.domain.lan\")"
time="2020-07-12T02:44:29+01:00" level=debug msg="Router reached with Host(\"adminer.domain.lan\") different from SNI(\"monitor.domain.lan\")"
time="2020-07-12T02:44:29+01:00" level=debug msg="Router reached with Host(\"adminer.domain.lan\") different from SNI(\"monitor.domain.lan\")"
time="2020-07-12T02:44:30+01:00" level=debug msg="Router reached with Host(\"pihole.domain.lan\") different from SNI(\"monitor.domain.lan\")"
time="2020-07-12T02:44:31+01:00" level=debug msg="Router reached with Host(\"adminer.domain.lan\") different from SNI(\"monitor.domain.lan\")"
time="2020-07-12T02:44:31+01:00" level=debug msg="Router reached with Host(\"adminer.domain.lan\") different from SNI(\"monitor.domain.lan\")"
time="2020-07-12T02:44:32+01:00" level=debug msg="Router reached with Host(\"pihole.domain.lan\") different from SNI(\"monitor.domain.lan\")"

time="2020-07-12T02:57:29+01:00" level=debug msg="Router reached with Host(\"pihole.domain.lan\") different from SNI(\"adminer.domain.lan\")"
time="2020-07-12T02:57:29+01:00" level=debug msg="Router reached with Host(\"pihole.domain.lan\") different from SNI(\"adminer.domain.lan\")"
time="2020-07-12T02:57:29+01:00" level=debug msg="Router reached with Host(\"pihole.domain.lan\") different from SNI(\"adminer.domain.lan\")"
time="2020-07-12T02:57:29+01:00" level=debug msg="Router reached with Host(\"pihole.domain.lan\") different from SNI(\"adminer.domain.lan\")"
time="2020-07-12T02:57:29+01:00" level=debug msg="Router reached with Host(\"pihole.domain.lan\") different from SNI(\"adminer.domain.lan\")"
time="2020-07-12T02:57:29+01:00" level=debug msg="Router reached with Host(\"pihole.domain.lan\") different from SNI(\"adminer.domain.lan\")"
time="2020-07-12T02:57:29+01:00" level=debug msg="Router reached with Host(\"pihole.domain.lan\") different from SNI(\"adminer.domain.lan\")"
time="2020-07-12T02:57:29+01:00" level=debug msg="Router reached with Host(\"pihole.domain.lan\") different from SNI(\"adminer.domain.lan\")"
time="2020-07-12T02:57:29+01:00" level=debug msg="Router reached with Host(\"pihole.domain.lan\") different from SNI(\"adminer.domain.lan\")"
time="2020-07-12T02:57:29+01:00" level=debug msg="Router reached with Host(\"pihole.domain.lan\") different from SNI(\"adminer.domain.lan\")"
time="2020-07-12T02:57:29+01:00" level=debug msg="Router reached with Host(\"pihole.domain.lan\") different from SNI(\"adminer.domain.lan\")"
time="2020-07-12T02:57:29+01:00" level=debug msg="Router reached with Host(\"whoami.domain.lan\") different from SNI(\"adminer.domain.lan\")"
time="2020-07-12T02:57:30+01:00" level=debug msg="Router reached with Host(\"whoami.domain.lan\") different from SNI(\"adminer.domain.lan\")"
time="2020-07-12T02:57:30+01:00" level=debug msg="Router reached with Host(\"pihole.domain.lan\") different from SNI(\"adminer.domain.lan\")"
time="2020-07-12T02:57:30+01:00" level=debug msg="Router reached with Host(\"portainer.domain.lan\") different from SNI(\"adminer.domain.lan\")"
time="2020-07-12T02:57:30+01:00" level=debug msg="Router reached with Host(\"portainer.domain.lan\") different from SNI(\"adminer.domain.lan\")"
time="2020-07-12T02:57:30+01:00" level=debug msg="Router reached with Host(\"portainer.domain.lan\") different from SNI(\"adminer.domain.lan\")"
time="2020-07-12T02:57:30+01:00" level=debug msg="Router reached with Host(\"grafana.domain.lan\") different from SNI(\"adminer.domain.lan\")"
time="2020-07-12T02:57:30+01:00" level=debug msg="Router reached with Host(\"grafana.domain.lan\") different from SNI(\"adminer.domain.lan\")"
time="2020-07-12T02:57:30+01:00" level=debug msg="Router reached with Host(\"grafana.domain.lan\") different from SNI(\"adminer.domain.lan\")"
time="2020-07-12T02:57:31+01:00" level=debug msg="Router reached with Host(\"pihole.domain.lan\") different from SNI(\"adminer.domain.lan\")"
time="2020-07-12T02:57:31+01:00" level=debug msg="Router reached with Host(\"grafana.domain.lan\") different from SNI(\"adminer.domain.lan\")"
time="2020-07-12T02:57:31+01:00" level=debug msg="Router reached with Host(\"portainer.domain.lan\") different from SNI(\"adminer.domain.lan\")"