HTTP Digest Auth & Continually Repeating Auth Prompts (on Chrome & IE, but not Safari)

I had a working Traefik 2.0 installation with HTTP Basic Auth for a few services with no issues across client browsers. I have upgraded the installation to use HTTP Digest Auth, and this is working fine via Safari which correctly sends saved credentials for sub-pages. However, on Chrome I get many repeated prompts for credentials even when I click to save them - if you have the patience everything works if you keep re-keying credentials but this is non-viable on a refreshing dashboard obviously.

Has anyone experienced this, and can they suggest a configuration which avoids this problem? In debugging this I have logged out of Chrome, cleared its cached data etc to ensure it wasn't saving/re-using any old HTTP Basic Auth credentials. Behaviour is same on Chrome whether on Mac or Windows. Similar behaviour also on IE (haven't tried Edge). I have also curl'ed the endpoints in verbose mode to see whether there are any returned HTTP headers that I should be propagating but as best as I can tell there are none with HTTP Digest Auth.

Obviously, I should be moving to a proper auth method...and keycloak does look like a good way for me to achieve this...but I would like to get this working as-is if possible.

2 Likes

Would you be able to start with posting your full configuration that reproduces the issue? If you do I could try and run it myself and see if I get the same issue, or if I can spot anything wrong with the configuration itself.

Additional config information based on @zespri's request

Also note that the Chrome app on iPad does not have this issue so it is seemingly very browser specific in its handling.

docker-compose.yml for Traefik

Enabling Traefik dashboard/apis and prometheus as sub-paths within main HTTPS site with DigestAuth enabled.

version: '3.3'
secrets:
  rsa_private_key:
    file: ${HOME}/.secrets/pdm/${ENV}/pdm-${ENV}.key.pem
  rsa_cert:
    file: ${HOME}/.secrets/pdm/${ENV}/pdm-${ENV}.crt.pem
configs:
  toml_conf:
    file: ../../config/${ENV}/traefik.toml
  dynamic_toml_conf:
    file: ../../config/dynamic_conf.toml
services:
  svc:
    # The official v2.0 Traefik docker image
    image: traefik:v2.0.2
    # Enables the web UI and tells Traefik to listen to docker
    ports:
      # Primary inbound HTTPS traffic.
      - "${PORT}:443"
      # HTTP traffic open for the purposes of permanent redirect to HTTPS.
      - "80:80"
    deploy:
      replicas: 3
      restart_policy:
        condition: on-failure
        max_attempts: 3
        delay: 30s
        window: 60s
      placement:
        constraints:
          - node.labels.pdm-core == yes
      labels:
        - "traefik.enable=true"
        # FIRST ROUTER - INSECURE REDIRECT
        - "traefik.http.routers.api.entrypoints=web"
        - "traefik.http.routers.api.rule=Host(`${ENV_FIRST_CHAR}-foo.bar`) && (PathPrefix(`/dashboard/`) || PathPrefix(`/api/`))"
        - "traefik.http.routers.api.middlewares=httpsredirect"
        - "traefik.http.routers.api.service=api@internal"
        # SECOND ROUTER - SECURE
        - "traefik.http.routers.api-sec.entrypoints=websecure"
        - "traefik.http.routers.api-sec.tls=true"
        - "traefik.http.routers.api-sec.tls.options=default"
        - "traefik.http.routers.api-sec.rule=Host(`${ENV_FIRST_CHAR}-foo.bar`) && (PathPrefix(`/dashboard/`) || PathPrefix(`/api/`))"
        - "traefik.http.routers.api-sec.service=traefik-dashboard-and-api"
        - "traefik.http.routers.api-sec.middlewares=auth"
        # THIRD ROUTER - PROMETHEUS SECURE
        - "traefik.http.routers.prom-sec.entrypoints=websecure"
        - "traefik.http.routers.prom-sec.tls=true"
        - "traefik.http.routers.prom-sec.tls.options=default"
        - "traefik.http.routers.prom-sec.rule=Host(`${ENV_FIRST_CHAR}-foo.bar`pdm) && PathPrefix(`/metrics`)"
        - "traefik.http.routers.prom-sec.service=prom"
        # MIDDLEWARES - AUTH & HTTPS REDIRECT
        # Note that any $'s in http digest need escaping with $$.
        - "traefik.http.middlewares.auth.digestauth.realm=pdm"
        - "traefik.http.middlewares.auth.digestauth.users=admin:pdm:<redacted>"
        - "traefik.http.middlewares.httpsredirect.redirectscheme.scheme=https"
        - "traefik.http.middlewares.httpsredirect.redirectscheme.permanent=true"
        - "traefik.http.middlewares.httpsredirect.redirectscheme.port=443"
        # Now the backend service itself...
        - "traefik.http.services.traefik-dashboard-and-api.loadbalancer.server.port=8080"
        - "traefik.http.services.prom.loadbalancer.server.port=8082"
    secrets:
      - source: rsa_private_key
        target: /etc/certs/server.key
      - source: rsa_cert
        target: /etc/certs/server.crt
    configs:
      - source: toml_conf
        target: /etc/traefik/traefik.toml
      - source: dynamic_toml_conf
        target: /etc/dynamic_conf.toml
    volumes:
      # So that Traefik can listen to the Docker events
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - default
      - other
    dns:
    - <Redacted>
networks:
  default:
    external:
      name: ${ENV}_spark
  other:
    external:
      name: ${ENV}_internal

Traefik.toml


################################################################
# Global configuration
################################################################

[global]
  checkNewVersion = true
  sendAnonymousUsage = false

################################################################
# Servers Transport
################################################################

[serversTransport]
  insecureSkipVerify = true

################################################################
# Entrypoints configuration
################################################################

[entryPoints]
  [entryPoints.web]
    address = ":80"

  [entryPoints.websecure]
    address = ":443"

  [entryPoints.metrics]
    address = ":8082"

################################################################
# Traefik logs configuration
################################################################

# Traefik logs
# Enabled by default and log to stdout

[log]
level = "WARN"

################################################################
# Access logs configuration
################################################################

[accessLog]
filePath = "/var/log/traefik_access.log"

################################################################
# Meterics configuration
################################################################

[metrics]
  [metrics.prometheus]
    buckets = [0.1,0.3,1.2,5.0]
    addEntryPointsLabels = true
    addServicesLabels = true
    entryPoint = "metrics"

################################################################
# API and dashboard configuration
################################################################

[api]
dashboard = true
insecure = true

################################################################
# Ping configuration
################################################################

[ping]

################################################################
# Docker configuration backend
################################################################

[providers]
  [providers.docker]
    exposedByDefault = false
    swarmMode = true
    network = "dev_internal"
  [providers.file]
    filename = "/etc/dynamic_conf.toml"



Hello @fifofonix , sorry for the delay. You also reference dynamic_conf.toml but I cannot see it posted. I'll try without it but may not arrive to the same results.

You specify providers.docker.network to be dev_internal and yet, this is the not the network, where your containers are on.

"traefik.http.routers.prom-sec.rule=Host(`${ENV_FIRST_CHAR}-foo.bar`pdm) && PathPrefix(/metrics)" - This is not a correct syntax pdm hanging after the apostrophe sign like that.

Yes, digest auth does not seem to work beyond simple "whoami" case.

Here is a minimal example:

version: "3.7"
services:
  traefik:
    image: traefik:v2.0.5
    ports:
      - "80:80"
      - "8080:8080"
    command:     
      - --api.insecure=true
      - --entryPoints.web.address=:80
      - --providers.docker.exposedByDefault=false
      - --log.level=DEBUG
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.dashboard.rule=PathPrefix(`/dashboard`)"
      - "traefik.http.routers.dashboard.service=dashboard"
      - "traefik.http.routers.dashboard.middlewares=auth"
      - "traefik.http.services.dashboard.loadbalancer.server.port=8080"
  whoami:
    image: containous/whoami
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=PathPrefix(`/`)"
      - "traefik.http.routers.whoami.entrypoints=web"
      - "traefik.http.routers.whoami.service=whoami"
      - "traefik.http.services.whoami.loadbalancer.server.port=80"
      - "traefik.http.routers.whoami.middlewares=auth"
      - "traefik.http.middlewares.auth.digestauth.realm=pdm"
      # Password: 1234
      - "traefik.http.middlewares.auth.digestauth.users=admin:pdm:c2cdbd1c7a9b2c9ff03f46f767a86edf"

For whoami it works. For the dashboard on chrome it floods you with never ending stream of authentication prompts. @ldez would you like to comment if we are doing anything wrong, before I open a github issue?

I see an existing, maybe related issue here: https://github.com/containous/traefik/issues/4281

There are also a couple of closed ones, with "cannot reproduce" comment:

https://github.com/containous/traefik/issues/4051
https://github.com/containous/traefik/issues/5596

First I highly discourage you to create a router on api when you are using --api.insecure.

The goal of the secure mode is to manage this kind of use case.

Also if you want to route the API with path only, you have to add /api to the routes.

Also the problem seems to appears only when there is no Host routing:

version: "3.7"
services:
  traefik:
    image: traefik:v2.0.5
    ports:
      - "80:80"
      - "8081:8081"
    command:     
      - --api
      - --entryPoints.web.address=:80
      - --entryPoints.api.address=:8081
      - --providers.docker.exposedByDefault=false
      - --log.level=DEBUG
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    labels:
      traefik.enable: true
      # traefik.http.routers.dashboard.rule: Host(`dashboard.localhost`)
      traefik.http.routers.dashboard.rule: PathPrefix(`/dashboard`) || PathPrefix(`/api`)
      traefik.http.routers.dashboard.entrypoints: api
      traefik.http.routers.dashboard.middlewares: auth
      traefik.http.routers.dashboard.service: api@internal

      # Password: 1234
      traefik.http.middlewares.auth.digestauth.realm: pdm
      traefik.http.middlewares.auth.digestauth.users: admin:pdm:c2cdbd1c7a9b2c9ff03f46f767a86edf

  whoami:
    image: containous/whoami
    labels:
      traefik.enable: true
      traefik.http.routers.whoami.rule: PathPrefix(`/`)
      # traefik.http.routers.whoami.rule: Host(`whoami.localhost`)
      traefik.http.routers.whoami.entrypoints: web
      traefik.http.routers.whoami.middlewares: auth
      

@ldez It is slightly better with your configuration when using host rules, however I'm still getting pop us every few seconds on chrome.

Do not worry about the router on the api - this is to model an external to traefik application, this is not a best practice guide. I wanted a minimal example, so instead of choosing an exteranl app to demonstrate that with, I used the dashboard you know very well. However I wanted to configure it the same way as I would configure any other application, that is without using the magic api@internal service.

I could have done spinning a second traefik instance with api@internal and then pointing the router in the main instance to a service pointing that that second instance, I wanted a setup to remain relatively simple though, so I have not done that.

Also if you want to route the API with path only, you have to add /api to the routes.

Yes, sorry, I missed that one. I tested with the /api too locally initially, and it also produced the pop ups, so I removed the /api. Now, when you pointed that out to me that the dashboard relies on the api, I'll make sure to keep it.

off-topic:

With 2.1, api@internal is now less magical, because we created a concrete internal provider, and the services, routers and middlewares of this provider now appear in the dashboard/api:

1 Like

ok but do you agree that the problem appears only when the routing is not based on Host rule?

No, in your example, uncommenting the host based rule and commenting the paths one I still get pop ups every few seconds in chrome.

I also changed whoami.localhost to whoami.internal and dashboard.localhost to dashboard.internal, since my desktop is windows and I'm running docker elsewhere - but that should hardly matter.

If I refresh in chrome, it always prompts for credentials several times. But even if I do not refresh, it still prompts every few seconds.

Here is "normal" operation, I'm not touching the keyboard except to enter the password prompts:

This is I refreshed the page:

traefik-digest-auth-02

Those brown 401s correspond to the prompts that I'm seeing.

The list of paths that generate those prompts is not always the same.

My latest config files, corrected according to what you said earlier:

docker-compose.yaml
version: "3.7"
services:
  traefik:
    image: traefik:v2.0.5
    ports:
      - "80:80"
      - "8080:8080"
    command:     
      - --api.insecure=true
      - --entryPoints.web.address=:80
      - --providers.docker.exposedByDefault=false
      - --log.level=DEBUG
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
  whoami:
    image: containous/whoami
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`whoami.internal`)"
      #- "traefik.http.routers.whoami.rule=PathPrefix(`/`)"
      - "traefik.http.routers.whoami.entrypoints=web"
      - "traefik.http.routers.whoami.service=whoami"
      - "traefik.http.services.whoami.loadbalancer.server.port=80"
      - "traefik.http.routers.whoami.middlewares=auth"
      # Password: 1234
      - "traefik.http.middlewares.auth.digestauth.realm=pdm"
      - "traefik.http.middlewares.auth.digestauth.users=admin:pdm:c2cdbd1c7a9b2c9ff03f46f767a86edf"
  traefik2:
    image: traefik:v2.0.5
    ports:
      - "8081:80"
    command:     
      - --api
      - --entryPoints.web.address=:80
      - --log.level=DEBUG
      - --providers.file.filename=/dyn.toml
    volumes:
      - "./dyn.toml:/dyn.toml"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.dashboard.rule=Host(`dashboard.internal`)"
      #- "traefik.http.routers.dashboard.rule=PathPrefix(`/dashboard`) || PathPrefix(`/api`)"
      - "traefik.http.routers.dashboard.service=dashboard"
      - "traefik.http.routers.dashboard.middlewares=auth"
      - "traefik.http.services.dashboard.loadbalancer.server.port=80"
dyn.toml
[http.routers.main]
entryPoints = ["web"]
service = "api@internal"
rule = "HostRegexp(`{host:.+}`)"

@ldez Please let me know if I can further help with diagnostics.

Example of /api/entrypoints that succeeds:

Request
GET http://dashboard.internal/api/entrypoints HTTP/1.1
Host: dashboard.internal
Connection: keep-alive
Authorization: Digest username="admin", realm="pdm", nonce="RL8D6d0qq6Yv7xoT", uri="/api/entrypoints", algorithm=MD5, response="4ef06bc43005700bf63b15f12b96c668", opaque="oKL8dRuAqqiDV0Oo", qop=auth, nc=00000009, cnonce="15fffc2e56652702"
Accept: application/json, text/plain, */*
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36
Referer: http://dashboard.internal/dashboard/
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9,ru;q=0.8
Response
HTTP/1.1 200 OK
Content-Length: 160
Content-Type: application/json
Date: Mon, 18 Nov 2019 03:30:41 GMT
X-Next-Page: 1

[{"address":":80","transport":{"lifeCycle":{"graceTimeOut":10000000000},"respondingTimeouts":{"idleTimeout":180000000000}},"forwardedHeaders":{},"name":"web"}]

Example of /api/etrypoints that asks for credentials straight after:

Request
GET http://dashboard.internal/api/entrypoints HTTP/1.1
Host: dashboard.internal
Connection: keep-alive
Authorization: Digest username="admin", realm="pdm", nonce="RL8D6d0qq6Yv7xoT", uri="/api/entrypoints", algorithm=MD5, response="355d4c6a6e921e9c40811256169306de", opaque="oKL8dRuAqqiDV0Oo", qop=auth, nc=0000000b, cnonce="33961d471ef75ad6"
Accept: application/json, text/plain, */*
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36
Referer: http://dashboard.internal/dashboard/
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9,ru;q=0.8
Response
HTTP/1.1 401 Unauthorized
Content-Type: text/plain
Www-Authenticate: Digest realm="pdm", nonce="ybc/9Jst+xGO/sPx", opaque="oKL8dRuAqqiDV0Oo", algorithm="MD5", qop="auth"
Date: Mon, 18 Nov 2019 03:30:46 GMT
Content-Length: 17

401 Unauthorized

Thanks for confirming replication of this issue and linking it on GitHub. Is it not bizarre that this works on Safari and even on iPad Chrome? Do we think this is actually therefore more of a browser compatibility/handling issue as opposed to a Traefik bug? Not isolated to desktop chrome however as also occurs on IE.

@ldez's input on secure dashboard also useful for me personally, as I now better understand the magic piece. Good to hear that is being addressed in future releases to increase transparency.

Same issue on Traefik 2.4.7 ,

simple webserver running labeled as :

    labels:
      traefik.enable: true
      traefik.http.routers.webserver.rule: Host(`webserver.${DOMAINNAME}`)
      traefik.http.routers.webserver.entrypoints: websecure
      traefik.http.services.webserver.loadbalancer.server.port: "${WEBSERVER_PORT:-8111}"
      traefik.http.routers.webserver.middlewares: auth

(same middlewares used for Traefik dashboad )
very surprising but see many reporting same issue since v1.7
Is there any WA for this?

@zespri, I meet the same problem and was unable to find a solution on the web (neither on stackoverflow or github).

Indeed I have Traefik in front of three applications, I have two applications that use digest-auth and the last one with no digest-auth. And when I restart the docker with no digest-auth, I always have a popup to ask me to login on the two others applications (that used digest-auth). This happens on Chrome and Mozilla, and I don't understand why each time I restart a docker all the containers with digest-auth "reset" the digest-auth ... Is there any solution ? Or maybe I misunderstanging the documentation and miss something ?

Thank you for your help :pray:

PS: I use traefik traefik:v2.4

I checked the Traefik logs, and I can see that each time I restart a docker that is expose via traefik but without middlewares, traefik recreates all the midllewares (and so all the digest-auth). Is there a way to restart a docker exposed on Traefik without recreating all the digest-auth middlewares ?

Is there any solution ?

I'm not aware of one

Is there a way to restart a docker exposed on Traefik without recreating all the digest-auth middlewares ?

It is technically impossible, it's not a traefik (or even docker) limitation, it's how process model in OS works.

Thank you for your feedback :slight_smile:

Bit late: for me the problem was I used both a Host match and a PathPrefix in the same rule. Since I use a dedicated subdomain for the dashboard anyway I simply removed the PathPrefix and now it also works in Google Chrome :slight_smile: