Traefik unable to get basic auth working despite using very basic password on docker swarm

I am unable to troubleshoot why basic auth does not work in my experimental setup. I am planning to use traefik with docker swarm and so far I have a few problems. This post is about understanding how to troubleshoot why the basic authentication fails. Notice i checked several other posts and also online on IRC with some other peeps, while the only suspect is the hash I am indeed using the apache encoding as documented and still keeps failing

My docker compose

version: '3'
services:
  traefik:
    image: traefik:v2.3
    ports:
      # The HTTP port
      - "80:80"
      # The Web UI (enabled by --api.insecure=true)
      # - "8080:8080"
      - "443:443"
    command:
      - "--log.level=DEBUG"
      - "--global.sendAnonymousUsage=true"
      # Docker swarm configuration
      - "--providers.docker"
      - "--providers.docker.endpoint=unix:///var/run/docker.sock"
      - "--providers.docker.swarmMode=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.docker.network=public"
      # Configure entrypoint
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.web.address=:80"
      # SSL configuration
      - "--certificatesresolvers.letsencryptresolver.acme.httpchallenge=true"
      - "--certificatesresolvers.letsencryptresolver.acme.httpchallenge.entrypoint=web"
      - "--certificatesresolvers.letsencryptresolver.acme.email=${LETS_ENCRYPT_EMAIL}"
      - "--certificatesresolvers.letsencryptresolver.acme.storage=/letsencrypt/acme.json"
      # Global HTTP -> HTTPS
      - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
      - "--entrypoints.web.http.redirections.entryPoint.scheme=https"
      # Enable dashboard/api
      - "--api.dashboard=true"
      - "--api.debug=true"
      - "--api.insecure=false"

    volumes:
      # To persist certificates
      - traefik-certificates:/letsencrypt
      # So that Traefik can listen to the Docker events
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - public
    deploy:
      placement:
        constraints:
          - node.role == manager
      labels:
        - "traefik.enable=true"
        - "traefik.http.services.traefik.loadbalancer.server.port=888" # required by swarm but not used.
        - "traefik.http.routers.traefik.rule=Host(`${TRAEFIK_URL}`) "
        - "traefik.http.routers.traefik.entrypoints=websecure"
        - "traefik.http.routers.traefik.tls.certresolver=letsencryptresolver"
        - "traefik.http.routers.traefik.service=api@internal"
        - "traefik.http.routers.traefik.middlewares=traefik-auth"
        - "traefik.http.middlewares.traefik-auth.basicauth.users=${TRAEFIK_BASIC_AUTH_USERS}"

volumes:
  traefik-certificates:
networks:
  public:
    external: true

How i initialise the service

LETS_ENCRYPT_EMAIL=awesomeEmail@bizmate.biz TRAEFIK_URL="traefik202101.amazingDomain.space"  TRAEFIK_BASIC_AUTH_USERS="admin:$$apr1$$kfY9ggNB$$DMuIKOae8f6VBU36bMCyW0"   DOCKER_HOST=ssh://swarm202101.amazingDomain.space docker stack deploy traefik -c traefik.yml
Creating service traefik_traefik

How i generate the hash, as you can see simple admin and password used, these are the literal values

echo $(htpasswd -nb admin password) | sed -e s/\\$/\\$\\$/g
admin:$$apr1$$kfY9ggNB$$DMuIKOae8f6VBU36bMCyW0

Now what i see when i open the traefik domain

If i open traefik domain i get

somehow the prompt is served by a non SSL page.

Then if i try to authenticate with admin and password i get

traefik_traefik.1.oook13dgppd6@amazigApp-infra-swarm-manager-0    | time="2021-02-02T09:54:31Z" level=debug msg="Authentication failed" middlewareName=traefik-auth@docker middlewareType=BasicAuth
traefik_traefik.1.oook13dgppd6@amazigApp-infra-swarm-manager-0    | time="2021-02-02T09:54:34Z" level=debug msg="Authentication failed" middlewareType=BasicAuth middlewareName=traefik-auth@docker

I can also see lets encrypt acme works and the basic auth is setup as such

traefik_traefik.1.oook13dgppd6@amazigApp-swarm-manager-0    | time="2021-02-02T09:57:30Z" level=debug msg="Configuration received from provider docker: {\"http\":{\"routers\":{\"traefik\":{\"entryPoints\":[\"websecure\"],\"middlewares\":[\"traefik-auth\"],\"service\":\"api@internal\",\"rule\":\"Host(`traefik202101.awesomedomain.space`) \",\"tls\":{\"certResolver\":\"letsencryptresolver\"}}},\"services\":{\"traefik\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.1.3:888\"}],\"passHostHeader\":true}}},\"middlewares\":{\"traefik-auth\":{\"basicAuth\":{\"users\":[\"admin:31235apr131235kfY9ggNB31235DMuIKOae8f6VBU36bMCyW0\"]}}}},\"tcp\":{},\"udp\":{}}" providerName=docker
traefik_traefik.1.oook13dgppd6@amazigApp-infra-swarm-manager-0    | time="2021-02-02T09:57:30Z" level=info msg="Skipping same configuration" providerName=docker

In the above i see the dollar sign shows as 31235 .... in other tests it also showed in logs as another set of numbers. But really i cannot figure out what is wrong. When i enter the user and password i get to the 401 error page and this is successfully served with SSL

I do not have anything special in between like other proxies and or other solutions that would strip and modify information/headers etc.
Any suggestions?

The full log from the service can be seen here Dropbox - traefik.logs - Simplify your life

Good job spotting this.

You're putting this in an environment variable? Are you defining it on the command line on in a .env(or other environment file)?
$$ in bash is the current PID. Your $$ is being evaluated as a shell argument between you defining it and the invocation of the container by docker-compose. If you used the value directly in the compose it would work.

If you really want to use an environment variable for this escaping like this \$$ should work.

The screengrab show https:// on the login dialog. Could you be experiencing some certificate error or another?

hi @cakiwi I am defining the variables in the command line as shown above. Never used the .env file before but I will indeed try it now.

About the certificate error, I am not 100% sure what is happening. Not sure if it is caching and as far as I can see in the logs the traefik domain SSL cert keeps getting regenerated (not sure if it is normal that it shows logs looking for the cert and generating it again and again) but I do not see an error for the SSL of traefik in logs. I see indeed the error on Chrome when i look in the icon of the address bar

I see it now. Have a look at the variable directly after you set it:

TRAEFIK_BASIC_AUTH_USERS="admin:$$apr1$$kfY9ggNB$$DMuIKOae8f6VBU36bMCyW0"  echo ${TRAEFIK_BASIC_AUTH_USERS}
admin:1552346apr11552346kfY9ggNB1552346DMuIKOae8f6VBU36bMCyW0

That is odd, at DEBUG I would expect to see some details. Including issuance if it succeeds.

Thank you so much for your time, I managed to fix the part to ensure the correct value is passed on when using environment variables. docker stack is not friendly with .env files as docker compose is so I am using the .env file but run
export $(cat .env) and the run the docker stack deployment and works better. I can see

traefik_traefik.1.og5yjk938rza@domain-infra-swarm-manager-0    | time="2021-02-02T13:28:44Z" level=debug msg="Configuration received from provider docker: {\"http\":{\"routers\":{\"traefik\":{\"entryPoints\":[\"websecure\"],\"middlewares\":[\"traefik-auth\"],\"service\":\"api@internal\",\"rule\":\"Host(`traefik202101.domain.space`) \",\"tls\":{\"certResolver\":\"letsencryptresolver\"}}},\"services\":{\"traefik\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.2.10:888\"}],\"passHostHeader\":true}}},\"middlewares\":{\"traefik-auth\":{\"basicAuth\":{\"users\":[\"admin:$$apr1$$kfY9ggNB$$DMuIKOae8f6VBU36bMCyW0\"]}}}},\"tcp\":{},\"udp\":{}}" providerName=docker

but still getting authentication errors. As you mentioned the certificate is generated, I think, here are some extracts from the last log that are relevant to ssl cert

traefik_traefik.1.scxu3lir42xh@myapp-infra-swarm-manager-0    | time="2021-02-02T15:45:47Z" level=debug msg="Adding certificate for domain(s) traefik202101.domain.space"
traefik_traefik.1.scxu3lir42xh@myapp-infra-swarm-manager-0    | time="2021-02-02T15:45:47Z" level=debug msg="No default certificate, generating one"
traefik_traefik.1.scxu3lir42xh@myapp-infra-swarm-manager-0    | time="2021-02-02T15:46:02Z" level=debug msg="Adding certificate for domain(s) traefik202101.domain.space"
traefik_traefik.1.scxu3lir42xh@myapp-infra-swarm-manager-0    | time="2021-02-02T15:46:02Z" level=debug msg="No default certificate, generating one"
....
traefik_traefik.1.scxu3lir42xh@myapp-infra-swarm-manager-0    | time="2021-02-02T15:46:02Z" level=debug msg="Adding route for traefik202101.domain.space with TLS options default" entryPointName=websecure
traefik_traefik.1.scxu3lir42xh@myapp-infra-swarm-manager-0    | time="2021-02-02T15:46:02Z" level=debug msg="Try to challenge certificate for domain [traefik202101.domain.space] found in HostSNI rule" providerName=letsencryptresolver.acme routerName=traefik@docker rule="Host(`traefik202101.domain.space`) "
traefik_traefik.1.scxu3lir42xh@myapp-infra-swarm-manager-0    | time="2021-02-02T15:46:02Z" level=debug msg="Looking for provided certificate(s) to validate [\"traefik202101.domain.space\"]..." providerName=letsencryptresolver.acme routerName=traefik@docker rule="Host(`traefik202101.domain.space`) "
traefik_traefik.1.scxu3lir42xh@myapp-infra-swarm-manager-0    | time="2021-02-02T15:46:02Z" level=debug msg="No ACME certificate generation required for domains [\"traefik202101.domain.space\"]." providerName=letsencryptresolver.acme routerName=traefik@docker rule="Host(`traefik202101.domain.space`) "

And I am still unable to sign in with basic auth and still see the homepage now secured

traefik_traefik.1.scxu3lir42xh@myapp-infra-swarm-manager-0    | time="2021-02-02T15:47:37Z" level=debug msg="Authentication failed" middlewareName=traefik-auth@docker middlewareType=BasicAuth
traefik_traefik.1.scxu3lir42xh@myapp-infra-swarm-manager-0    | time="2021-02-02T15:47:45Z" level=debug msg="Authentication failed" middlewareName=traefik-auth@docker middlewareType=BasicAuth

the full log is here ... I really cannot figure out anything else from this

I almost forgot. I am still seeing the basic auth prompt served with an incorrect certificate but the ssl looks fine when on the 401 error page.

What other information can I look at to fix both the authentication not working and the SSL serving on the homepage of the authentication page?
I wonder that the authentication might just not be working because of how it is being served.

Hi, I'm very new to traefik and i'm having a similar problem and I have no Idea what's wrong. I basically just followed the documentation here BasicAuth - Traefik | Site | v2.0.. everything except auth is working fine. If I remove the auth middleware my dashboard and other services are showing up just like they should. the log just says "authentication failed".. may I ask for help? :slight_smile:

traefik    | time="2021-09-13T20:20:12+02:00" level=debug msg="Authentication fa
iled" middlewareName=traefik-auth@docker middlewareType=BasicAuth

my docker-compose.yml

version: '3'

services:
  traefik:
    image: traefik:latest
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - proxy
    ports:
      - 80:80
      - 443:443
    environment:
      - CF_API_EMAIL=xxx
      - CF_API_KEY=xxx
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./data/traefik.yml:/traefik.yml:ro
      - ./data/acme.json:/acme.json
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`traefik.example.com`)"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=user:$$apr1$$..."
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`traefik.example.com`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=cloudflare"
      - "traefik.http.routers.traefik-secure.tls.domains[0].main=example.com"
      - "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.example.com"
      - "traefik.http.routers.traefik-secure.service=api@internal"
networks:
  proxy:
    external: true

my traefik.yml

api:
  dashboard: true
  debug: true

entryPoints:
  http:
    address: ":80"
  https:
    address: ":443"

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false

certificatesResolvers:
  cloudflare:
    acme:
      email: mail@example.com
      storage: acme.json
      dnsChallenge:
        provider: cloudflare
        resolvers:
          - "1.1.1.1:53"
          - "1.0.0.1:53"

# Writing Logs to a File
log:
  level: DEBUG

Hello.

  1. i not recommend use volumes in docker swarm.
  2. put your files on shared storage for docker swarm (nfs)
    take a look at Traefik 2.5.2 TLS 1.0 1.1 disable

/external/efs/ - AWS EFS service (unix nfs) folder services available on all swarm nodes with same content.

Thanks for your reply. Unlike the OP i'm not using swarm mode. Since I don't have any issues running traefik and other services in general I doubt that this is the problem. Only thing which isn't working for me is basic auth :confused:

Basic auth might not work because of

  • by using the inline option you are not encoding it correctly and thus the value in the container is incorrect
  • the version of htpasswd you use generates the passwords using a hash that is not compatible. I use an online website now to generate them but you can most likely force a specific algorithm like bcrypt when generating the password
  • consider using a file for the passwords so you are not wondering about escaping characters

About clusters and volumes i use s3fs-fuse for mounting S3 buckets as volumes - Ansible Galaxy

1 Like

thank you! I put the password in a password file and didnt use htpasswd and it worked