Nothing works so where to start?

[roelofz]

Hi,

After being warned in numerous pages and video's I took a few weeks to prepare my implementation and create a docker-compose-override. I freed ports on the existing services, prepared everything and the result was a frustrating day trying to get it to work, so in short (with a tiny voice): help...

Here's my docker compose override, the best I could configure without failure notices:

version: '2.1'
networks:
frontend:
name: frontend
services:
traefik:
image: traefik:v2.10.7
container_name: traefik
restart: unless-stopped
command:
- "--api=true"
- "--api.dashboard=true"
- "--log.level=DEBUG"
- "--entrypoints.imap.adress=:993"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--certificatesresolvers.le.acme.email=info@domain.nl"
- "--certificatesresolvers.le.acme.storage=/cert/acme.json"
- "--certificatesresolvers.le.acme.tlschallenge=true"
- "--entrypoints.web.address=:80"
- "--entrypoints.websecure.address=:443"
- "--entrypoints.web.http.redirections.entryPoint.to=websecure"
- "--entrypoints.web.http.redirections.entryPoint.scheme=https"
ports:
- "80:80"
- "443:443"
- "993:993"
- "8090:8080"
networks:
- frontend
labels:
- traefik.enable=true
- traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https
- traefik.http.routers.redirs.rule=hostregexp({host:.+})
- traefik.http.routers.redirs.entrypoints=web
- traefik.http.routers.redirs.middlewares=redirect-to-https
- traefik.http.routers.traefik_https.rule=Host(traefik.domain.nl)
- traefik.http.routers.traefik_https.entrypoints=websecure
- traefik.http.routers.traefik_https.tls=true
- traefik.http.routers.traefik_https.tls.certresolver=le
- traefik.http.routers.traefik_https.service=api@internal
- traefik.http.routers.traefik_https.middlewares=basic-auth-global
- traefik.http.middlewares.admin.basicauth.users=user:hash
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- acme:/cert
- /opt/traefik:/etc/traefik

Here's my traefik.yml:

log:
level: debug
filePath: "/etc/traefik/traefik.log"
accesslog:
filepath: "/etc/traefik/access.log"

So what's the result:

• Dasboard page http 404, https not found
• No certs are drawn
• no other application is passed through (not mentioned as the above should be enough to work on, if this works, I get the rest working).

Snip from access log:
34.219.213.94 - - [26/Dec/2023:23:00:05 +0000] "GET /.well-known/acme-challenge/qsZNXwQdmRf6P-b5rTIM-iUXmzgTlTiftLMI_7UeLOs HTTP/1.1" 404 19 "-" "-" 3 "-" "-" 0ms
Snip from Traefik.log:
time="2023-12-26T22:54:44Z" level=info msg="Starting provider *traefik.Provider"
time="2023-12-26T22:54:44Z" level=debug msg="*traefik.Provider provider configuration: {}"
time="2023-12-26T22:54:44Z" level=info msg="Starting provider *acme.ChallengeTLSALPN"
time="2023-12-26T22:54:44Z" level=debug msg="*acme.ChallengeTLSALPN provider configuration: {}"
time="2023-12-26T22:54:44Z" level=debug msg="Configuration received: {"http":{"services":{"noop":{}},"serversTransports":{"default":{"maxIdleConnsPerHost":200}}},"tcp":{},"udp":{},"tls">
time="2023-12-26T22:54:44Z" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
time="2023-12-26T23:02:29Z" level=info msg="I have to go..."
time="2023-12-26T23:02:29Z" level=info msg="Stopping server gracefully"
time="2023-12-26T23:02:29Z" level=debug msg="Waiting 10s seconds before killing connections." entryPointName=http
time="2023-12-26T23:02:29Z" level=error msg="accept tcp [::]:80: use of closed network connection" entryPointName=http
time="2023-12-26T23:02:29Z" level=error msg="close tcp [::]:80: use of closed network connection" entryPointName=http
time="2023-12-26T23:02:29Z" level=debug msg="Entry point http closed" entryPointName=http
time="2023-12-26T23:02:29Z" level=info msg="Server stopped"
time="2023-12-26T23:02:29Z" level=info msg="Shutting down"

So the only result for me is that there are no failures mentioned, I wish there was more to work with but I can not find any logging helping me more.

So where do I start, what setting is activated, what is not, what do I need to change.

Thanks for reading this far, hope you can help me!

[bluepuma77]

Use 3 backticks in front and after code, to make it more readable. And in yaml every space counts.

Check and compare to simple Traefik example.

[roelofz]

Found the backticks.... (pity I could not edit the first post anymore)
I've checked numerous times if the spaces were in order, could not find the mistake.
For me, this looks correct, after all I have read (and tryed).

docker-compose-override.yml

version: '2.1' 
networks:
    frontend:
      name: frontend

services:
    traefik:
      image: traefik:v2.10.7
      container_name: traefik
      restart: unless-stopped
      command:
        - "--api=true"
        - "--api.dashboard=true"
        - "--log.level=DEBUG"
        - "--entrypoints.imap.adress=:993"
        - "--providers.docker=true"
        - "--providers.docker.exposedbydefault=false"
        - "--certificatesresolvers.le.acme.email=info@site.nl"
        - "--certificatesresolvers.le.acme.storage=/cert/acme.json"
        - "--certificatesresolvers.le.acme.tlschallenge=true"
        - "--entrypoints.web.address=:80"
        - "--entrypoints.websecure.address=:443"
        - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
        - "--entrypoints.web.http.redirections.entryPoint.scheme=https"
      ports:
        - "80:80"
        - "443:443"
        - "993:993"
        - "8090:8080"
      networks:
        - frontend

      labels:
        - traefik.enable=true
        - traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https
        - traefik.http.routers.redirs.rule=hostregexp(`{host:.+}`)
        - traefik.http.routers.redirs.entrypoints=web
        - traefik.http.routers.redirs.middlewares=redirect-to-https
        - traefik.http.routers.traefik_https.rule=Host(`traefik.site.nl`)
        - traefik.http.routers.traefik_https.entrypoints=websecure
        - traefik.http.routers.traefik_https.tls=true
        - traefik.http.routers.traefik_https.tls.certresolver=le
        - traefik.http.routers.traefik_https.service=api@internal
        - traefik.http.routers.traefik_https.middlewares=basic-auth-global
        - traefik.http.middlewares.admin.basicauth.users=user:hash

      volumes:
        - /var/run/docker.sock:/var/run/docker.sock
        - acme:/cert
        - /opt/traefik:/etc/traefik

Traefik.yml:

log:
  level: debug
  filePath: "/etc/traefik/traefik.log"
accesslog:
  filepath:  "/etc/traefik/access.log"

[bluepuma77]

You can not mix Traefik static config. Use traefik.yml or command.

[roelofz]

What has to be in Trafik.yml and should not be in docker-compose?
Only one I can think of now is the log.level notation
This was puzzling, reading the docs too. It is not that clear where to put what (everything is described as in cli, yml and toml, so that does not indicate where to put it).

I am really puzzled where to put what, so hope you can help me with that.

[roelofz]

Or... should I put all commands in Traefik.yml in docker-compose suing the command option?
How to translate these two file paths in commands?
it almost seems too simple if this is the solution.

[bluepuma77]

You can only use one static config. There is a priority list, first one found is used, the rest is ignored.

As stated before, check simple Traefik example, it’s all there. Or check Traefik doc how to translate your parameters between formats.

[roelofz]

All combined into commands and found one error in the IMAP declaration, all failed because off that:

version: '2.1' 
networks:
    frontend:
      name: frontend

services:
    traefik:
      image: traefik:v2.10.7
      container_name: traefik
      restart: unless-stopped
      command:
        - "--api=true"
        - "--api.dashboard=true"
        - "--log.level=DEBUG"
        - "--entrypoints.imap**.adress=**:993"
        - "--providers.docker=true"
        - "--providers.docker.exposedbydefault=false"
        - "--certificatesresolvers.le.acme.email=info@site.nl"
        - "--certificatesresolvers.le.acme.storage=/cert/acme.json"
        - "--certificatesresolvers.le.acme.tlschallenge=true"
        - "--entrypoints.web.address=:80"
        - "--entrypoints.websecure.address=:443"
        - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
        - "--entrypoints.web.http.redirections.entryPoint.scheme=https"
      ports:
        - "80:80"
        - "443:443"
        - "993:993"
        - "8090:8080"
      networks:
        - frontend

      labels:
        - traefik.enable=true
        - traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https
        - traefik.http.routers.redirs.rule=hostregexp(`{host:.+}`)
        - traefik.http.routers.redirs.entrypoints=web
        - traefik.http.routers.redirs.middlewares=redirect-to-https
        - traefik.http.routers.traefik_https.rule=Host(`traefik.site.nl`)
        - traefik.http.routers.traefik_https.entrypoints=websecure
        - traefik.http.routers.traefik_https.tls=true
        - traefik.http.routers.traefik_https.tls.certresolver=le
        - traefik.http.routers.traefik_https.service=api@internal
        - traefik.http.routers.traefik_https.middlewares=basic-auth-global
        - traefik.http.middlewares.admin.basicauth.users=user:hash

      volumes:
        - /var/run/docker.sock:/var/run/docker.sock
        - acme:/cert
        - /opt/traefik:/etc/traefik

Now the dasboard is the only one not running, all other services do, need to find what is missing there...
I can not use 8080, so changed it to 8090:8080, but now not sure how to connect it
Added this one:

         - traefik.http.services.traefik_https.loadbalancer.server.port=8090

But this seems not enough (tried both ports).

[bluepuma77]

You don’t create an entrypoint on 8080, so Traefik is not listening.

Traefik only creates 8080 automatically, when using insecure dashboard. But with that all middlewares is skipped, no auth is possible.

It’s all explained in doc and in the simple Traefik example.

[roelofz]

The documentation may seem logical to you, to me itś being thrown around from option to option, no complete examples exist in the documentation.
This is from the simple Traefik example, no word about 8080:

      #- --accesslog.filepath=/var/log/traefik-access.log
      - --providers.docker.network=proxy
      - --providers.docker.exposedByDefault=false
      - --entrypoints.web.address=:80
      - --entrypoints.web.http.redirections.entrypoint.to=websecure
      - --entryPoints.web.http.redirections.entrypoint.scheme=https
      - --entrypoints.websecure.address=:443
      - --entrypoints.websecure.asDefault=true
      - --entrypoints.websecure.http.tls.certresolver=myresolver
      - --certificatesresolvers.myresolver.acme.email=mail@example.com
      - --certificatesresolvers.myresolver.acme.tlschallenge=true
      - --certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json
    labels:
      - traefik.enable=true
      - traefik.http.routers.mydashboard.rule=Host(`traefik.example.com`)
      - traefik.http.routers.mydashboard.service=api@internal
      - traefik.http.routers.mydashboard.middlewares=myauth
      - traefik.http.middlewares.myauth.basicauth.users=test:$$apr1$$H6uskkkW$$IgXLP6ewTrSuBkTrqE8wj/

So how is the example helping me as it does not mention what I am seeking or you are mentioning?

I don´t want to use 8080 as I want to use 443, all declared correctly when looking at my other sites.
For my other sites I had to switch the original ports (443) to 8443, same with the 80 port.
So I assume something simular needs to be done for the dashboard, but I don´t know...

So I must be missing something here, must have declared to much labels, who knows....

Current config:

version: '2.1'
networks:
    frontend:
#      attachable: true
      name: frontend

services:
    traefik:
      image: traefik:v2.10.7
      container_name: traefik
      restart: unless-stopped
      command:
        - "--api=true"
        - "--api.dashboard=true"
        - "--accesslog=true"
        - "--accesslog.filepath=/etc/traefik/access.log"
        - "--log.filePath=/etc/traefik/traefik.log"
        - "--log.level=DEBUG"
        - "--entrypoints.imap.address=:993"
        - "--providers.docker=true"
        - "--providers.docker.exposedbydefault=false"
        - "--certificatesresolvers.le.acme.email=info@mysite.nl"
        - "--certificatesresolvers.le.acme.storage=/cert/acme.json"
        - "--certificatesresolvers.le.acme.tlschallenge=true"
        - "--entrypoints.web.address=:80"
        - "--entrypoints.websecure.address=:443"
        - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
        - "--entrypoints.web.http.redirections.entryPoint.scheme=https"
      ports:
        - "80:80"
        - "443:443"
        - "993:993"
#        - "8090:8080" (needed or not?)
      networks:
        frontend:
      labels:
        - traefik.enable=true

        - traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https
        - traefik.http.routers.redirs.rule=hostregexp(`{host:.+}`)
        - traefik.http.routers.redirs.entrypoints=web
        - traefik.http.routers.redirs.middlewares=redirect-to-https
        - traefik.http.routers.api.rule=Host(`traefik.mysite.nl`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`)
        - traefik.http.routers.api.entrypoints=websecure
        - traefik.http.routers.api.tls=true
        - traefik.http.routers.api.tls.certresolver=le
        - traefik.http.routers.api.service=api@internal
        - traefik.http.routers.api.middlewares=basic-auth-global
        - traefik.http.middlewares.api.basicauth.users=user:hash
      volumes:
        - /var/run/docker.sock:/var/run/docker.sock
        - acme:/cert
        - /opt/traefik:/etc/traefik

[bluepuma77]

To be more clear, the example places dashboard on 443, as that port supports easy LetsEncrypt.

• Traefik dashboard is enabled at https://traefik.example.com/dashboard/ with user/pass test/test

(default https port 443)

Note when using basicAuth in labels, you need to escape every $ with another $.

[roelofz]

I changed as follows:

version: '2.1'
networks:
    frontend:
      name: frontend
      driver: overlay
      external: true

services:
    traefik:
      image: traefik:v2.10.7
      container_name: traefik
      restart: unless-stopped
      command:
        - "--api=true"
        - "--api.dashboard=true"
        - "--accesslog=true"
        - "--accesslog.filepath=/etc/traefik/access.log"
        - "--log.filePath=/etc/traefik/traefik.log"
        - "--log.level=DEBUG"
        - "--entrypoints.imap.address=:993"
        - "--providers.docker=true"
        - "--providers.docker.exposedbydefault=false"
        - "--certificatesresolvers.le.acme.email=info@nijenzee.nl"
        - "--certificatesresolvers.le.acme.storage=/cert/acme.json"
        - "--certificatesresolvers.le.acme.tlschallenge=true"
        - "--entrypoints.web.address=:80"
        - "--entrypoints.websecure.address=:443"
        - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
        - "--entrypoints.web.http.redirections.entryPoint.scheme=https"
      ports:
        - "80:80"
        - "443:443"
        - "993:993"
      networks:
        frontend:
        mailcow-network:
      labels:
        - traefik.enable=true
        - traefik.http.routers.mydashboard.rule=Host(`traefik.nijenzee.nl`)
        - traefik.http.routers.mydashboard.service=api@internal
        - traefik.http.routers.mydashboard.middlewares=myauth
        - traefik.http.middlewares.myauth.basicauth.users=adm:$$2y$$05$$Y2kZnsoy2EBybURNPb9T0OmCLGquZ/566zdpM.C50Wp5F1vCcGc3W
      volumes:
        - /var/run/docker.sock:/var/run/docker.sock
        - acme:/cert
        - /opt/traefik:/etc/traefik

I simplified hopefully the right way, removed a lot to comply with the example. I changed the network definition hoping this may help (reverted, external and overlay is not allowed when existing, and can not be created in docker compose), still no dice for the dashboard. Where should I expect it, at the domain or with /dashboard (testing both)?

I assume you mean using variables in labels, otherwise I don´t understand what you mean.

[bluepuma77]

Looks good, should be available at https://traefik.nijenzee.nl/dashboard/. If not, check Traefik debug log for errors (doc).

[roelofz]

"'502 Bad Gateway' caused by: dial tcp 172.18.0.5:8080: connect: connection refused"

This one seems to be related.

I earlier searched for this one, could only find a mixup in configuration files, not my case anymore.

[bluepuma77]

I doubt this is related to dashboard. You have other Docker services/containers running or any dynamic config files? Using port 8080 or having expose 8080 in Dockerfile.

Usually "Bad gateway" happens when a target service/container is attached to multiple network and not all are shared with Traefik. Use docker.network= to set the dedicated network to use.

Note that Docker compose usually changes the network name (prepends project name), so make sure to use an external one or explicitly set name. Or adapt the label docker.network= to the project network name.

[roelofz]

I did not want to flood post my complete config, but you are more then right: I'm running Mailcow (a mailserver), Nextcloud (inside mailcow and reason for Traefik, it needs to be outside Mailcow's Nginx).
I think I have a more then complicated puzzle right now, but I'm beginning to see some light. Mailcow is running fine now, so step by step I'm getting there.

Thanks for your help, I will update when all is running as it should run...

[roelofz]

I isolated the config, shutting down all what is running and I got the dashboard running, after discovering the users line is breaking the config. It is impossible to get a working prompt, using the hashed password.
I am almost sure, the website is not broken by my services but by the password config (can´t set this live, leaving an open dashboard site).

version: '2.1' networks:
    frontend:
      name: frontend

services:
    traefik:
      image: traefik:v2.10.7
      container_name: traefik
      restart: unless-stopped
      command:
        - "--api=true"
        - "--api.dashboard=true"
        - "--api.debug=true"
        - "--accesslog=true"
        - "--accesslog.filepath=/etc/traefik/access.log"
        - "--log.filePath=/etc/traefik/traefik.log"
        - "--log.level=DEBUG"
        - "--providers.docker=true"
        - "--providers.docker.exposedbydefault=false"
        - "--certificatesresolvers.le.acme.email=info@site.nl"
        - "--certificatesresolvers.le.acme.storage=/cert/acme.json"
        - "--certificatesresolvers.le.acme.tlschallenge=true"
        - "--entrypoints.web.address=:80"
        - "--entrypoints.websecure.address=:443"
        - "--entrypoints.imap.address=:993"
        - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
        - "--entrypoints.web.http.redirections.entryPoint.scheme=https"
      ports:
        - "80:80"
        - "443:443"
        - "993:993"
      networks:
        frontend:
      labels:
        - traefik.enable=true
        - traefik.http.routers.mydashboard.rule=Host(`traefik.site.nl`)
        - traefik.http.routers.mydashboard.tls=true
        - traefik.http.routers.mydashboard.service=api@internal
        - traefik.http.middlewares.myauth.basicauth.usersfile=/etc/traefik/log.pss
        - traefik.http.routers.mydashboard.middlewares=myauth
      volumes:
        - /var/run/docker.sock:/var/run/docker.sock
        - acme:/cert
        - /opt/traefik:/etc/traefik
volumes:
  acme:
    driver: local
    driver_opts:
      type: 'none'

I used this command to create the hash:
echo $(htpasswd -nb user password) | sed -e s/\$/\$\$/
I am seeing dollars now, added, removed, but it still isn´t working.
Tried first without a password file.
So I get in the dashboard, but without a working popup, loggine is as follows:

level=debug msg="Authentication succeeded" middlewareName=myauth@docker middlewareType=BasicAuth

So how to tackle this password issue?

[bluepuma77]

If it says auth succeeded, then I would assume you (or your browser) delivered user/pass.

Check browsers developer tools network tab (its encrypted in header) or just change the password to test.

[roelofz]

Hi and best wishes for the new year!

Auth succeeds without a prompt, this happens with the hashed password, without the sed command:
echo $(htpasswd -nb user verylongpassword) | sed -e s/\$/\$\$/g
With the sed command, the prompt is there, but cannot verify the password (auth failed in the logging).
I disabled characters like ! in the password gen (very secure to loose these, but all for testing).

Is this expected behaviour?

[bluepuma77]

Just try with simple Traefik example auth string.

      - traefik.http.routers.mydashboard.middlewares=myauth
      - traefik.http.middlewares.myauth.basicauth.users=test:$$apr1$$H6uskkkW$$IgXLP6ewTrSuBkTrqE8wj/

User: test
Pass: test